This recipe shows you how to follow a subject across multiple videos and build a chronological timeline of appearances. The response includes timestamps, video references, locations, and context for each sighting, ready for your tracking dashboard or analysis pipeline.

Use cases:

• Security investigations: Track a person across multiple camera angles or footage sources
• Timeline construction: Build a chronological record of a subject’s appearances
• Content auditing: Identify every moment where a specific entity appears in a collection

Key concepts

• Knowledge store: A persistent store of your videos and images plus the understanding the platform derives from them — spatiotemporal context, a typed ontology, and embeddings — that together enable corpus-level reasoning.
• Instructions: Additional guidance that shapes Jockey’s behavior for a specific domain or task.
• Structured output: A JSON schema you provide with your request. Jockey constrains its output to match your schema, so you retrieve machine-readable results instead of plain text. For a guide on designing schemas and parsing responses, see the Structured output page.

Workflow

This recipe combines three elements: a JSON schema that defines the appearance data structure, instructions that give Jockey a specific role and domain guidance, and a prompt that describes the subject to track. Jockey reasons across your knowledge store and returns structured tracking results. You can then continue the conversation using the session identifier from the response. For example, drill into a specific appearance or ask about the surrounding context. You can adapt this recipe to track different subjects or focus on different appearance details.

Prerequisites

• You’ve already created a knowledge store with at least one item in ready status. See the Quickstart page for details.
• You’re familiar with the request and response format. See the Create a response page for details.

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

1
import json
2
from twelvelabs import TwelveLabs, TextParam
3
from twelvelabs.types.text_param_format import TextParamFormat_JsonSchema
4
5
client = TwelveLabs(api_key="<YOUR_API_KEY>")
6
STORE_ID = "<YOUR_KNOWLEDGE_STORE_ID>"
7
8
# Define a JSON schema for the tracking timeline
9
tracking_schema = {
10
    "type": "object",
11
    "properties": {
12
        "subject": {"type": "string"},              # Description of the tracked subject
13
        "timeline": {
14
            "type": "array",
15
            "items": {
16
                "type": "object",
17
                "properties": {
18
                    "timestamp": {"type": "string"},        # When the appearance occurs
19
                    "video_reference": {"type": "string"},  # Source video identifier
20
                    "location": {"type": "string"},         # Where in the scene
21
                    "action": {"type": "string"},           # What the subject is doing
22
                },
23
            },
24
        },
25
        "summary": {"type": "string"},              # Overall tracking summary
26
    },
27
}
28
29
# Track the subject
30
response = client.responses.create(
31
    knowledge_store_id=STORE_ID,
32
    instructions="You are a security analyst. Prioritize temporal accuracy. Flag any low-confidence identifications.",
33
    input=[{"type": "message", "role": "user", "content": "Track the person in the blue hoodie across all cameras. Give me a chronological timeline."}],
34
    text=TextParam(format=TextParamFormat_JsonSchema(name="entity_tracking", schema_=tracking_schema)),
35
)
36
37
# Parse and read the timeline
38
for output in response.output:
39
    if output.type == "message":
40
        for content in output.content:
41
            data = json.loads(content.text)
42
            print(f"Subject: {data['subject']}")
43
            print(f"Summary: {data['summary']}\n")
44
            for entry in data["timeline"]:
45
                print(f"  [{entry['timestamp']}] {entry['video_reference']}")
46
                print(f"    Location: {entry['location']}")
47
                print(f"    Action: {entry['action']}")
48
49
# Refine with a follow-up turn
50
follow_up = client.responses.create(
51
    knowledge_store_id=STORE_ID,
52
    session_id=response.session_id,
53
    input=[{"type": "message", "role": "user", "content": "Tell me more about what happened at the third timestamp. What was happening around the subject?"}],
54
)
55
56
# Read the follow-up response
57
for output in follow_up.output:
58
    if output.type == "message":
59
        for content in output.content:
60
            print(content.text)

Code explanation

Example response

The text field inside each content part is a JSON string that matches your schema. After parsing, a typical result looks like this:

1
{
2
  "subject": "Person in blue hoodie",
3
  "timeline": [
4
    {
5
      "timestamp": "02:15-03:40",
6
      "video_reference": "069eb4e8-aeb0-7e83-8000-86413fcc296a",
7
      "location": "Building lobby, near the front entrance",
8
      "action": "Enters through the main door and walks toward the elevator"
9
    },
10
    {
11
      "timestamp": "05:42-06:10",
12
      "video_reference": "069e1e97-27f8-7a8e-8000-bf32ecd7fc8c",
13
      "location": "Third floor hallway",
14
      "action": "Exits the elevator and walks down the corridor"
15
    }
16
  ],
17
  "summary": "Subject appears in 2 videos over a 3-minute span, moving from the lobby to the third floor."
18
}

Variations

Change the instructions and prompt to adapt this recipe for different tracking scenarios.

• Change the analytical perspective: Set the instructions parameter to “documentary researcher” or “video editor” for different emphasis.
• Track objects instead of people: Change the prompt to track a vehicle, branded object, or recurring visual element.
• Multi-episode tracking: Track a recurring character or subject across an episode series.

See also

• Extract entities - list all entities before deciding which to track
• Structured output - more on JSON Schema responses
• Multi-turn sessions - more on continuing conversations
• Create a response - API reference for the Responses endpoint