Use the TwelveLabs Video Understanding Platform to find specific moments in your video content using natural language queries or reference images. The platform analyzes videos by integrating images, audio, speech, and text, offering a deeper understanding than single-modal methods. It captures complex relationships between these elements, detects subtle details, and supports natural language queries and images for intuitive and precise use.

Key features:

• Improved accuracy: Multimodal integration enhances accuracy.
• Easy interaction: Natural language queries simplify searches.
• Advanced search: Enables image-based queries for precise results.
• Fewer errors: Multi-faceted analysis reduces misinterpretation.
• Time savings: Quickly finds relevant clips without manual review.

Use cases:

• Spoken word search: Find video segments where specific words or phrases are spoken.
• Visual element search: Locate video segments that match descriptions of visual elements or scenes.
• Action or event search: Identify video segments that depict specific actions or events.
• Image similarity search: Find video segments that visually resemble a provided image.
• Entity search: Locate video segments containing specific people, car models, animal species, or branded objects with improved accuracy (Marengo 3.0 only).

The platform supports the following types of search queries:

• Text queries: Search using natural language descriptions of visual elements, actions, sounds, or spoken words.
• Image queries: Search using images to find visually similar content in your videos.
• Composed queries: Combine text descriptions with images for more precise results (Marengo 3.0 only).

For guidance on choosing the correct type of query for your use case, see Search with text, image, and composed queries.

To understand how your usage is measured and billed, see the Pricing page.

Prerequisites

• To use the platform, you need an API key:

  1

  If you don’t have an account, sign up for a free account.

  2

  Go to the API Keys page.

  3

  Select the Copy icon next to your key.

• Depending on the programming language you are using, install the TwelveLabs SDK by entering one of the following commands:

  PythonNode.js

  $
  pip install twelvelabs

• Your video files must meet the format requirements.

• If you wish to use images as queries, ensure that your image file meet the format requirements.

Complete examples

Thesecomplete examples shows how to create an index, upload a video, and perform search requests using text, image, and composed queries. Ensure you replace the placeholders surrounded by <> with your values.

1
from twelvelabs import TwelveLabs
2
from twelvelabs.indexes import IndexesCreateRequestModelsItem
3
from twelvelabs.tasks import TasksRetrieveResponse
4
5
# 1. Initialize the client
6
client = TwelveLabs(api_key="<YOUR_API_KEY>")
7
8
# 2. Create an index
9
index = client.indexes.create(
10
    index_name="<YOUR_INDEX_NAME>",
11
    models=[
12
        IndexesCreateRequestModelsItem(
13
            model_name="marengo3.0",
14
            model_options=["visual", "audio"]
15
        )
16
    ]
17
)
18
print(f"Created index: id={index.id}")
19
20
# 3. Upload a video
21
task = client.tasks.create(
22
    index_id=index.id, video_url="<YOUR_VIDEO_URL>")
23
print(f"Created task: id={task.id}")
24
25
# 4. Monitor the indexing process
26
def on_task_update(task: TasksRetrieveResponse):
27
    print(f"  Status={task.status}")
28
29
task = client.tasks.wait_for_done(sleep_interval= 5, task_id=task.id, callback=on_task_update)
30
if task.status != "ready":
31
    raise RuntimeError(f"Indexing failed with status {task.status}")
32
print(
33
    f"Upload complete. The unique identifier of your video is {task.video_id}.")
34
35
# 5. Perform a search request
36
search_pager = client.search.query(
37
    index_id=index.id,
38
    query_text="<YOUR_QUERY>",
39
    search_options=["visual", "audio"],
40
    # operator="or" # Optional: Use "and" to find segments matching all modalities
41
    # transcription_options=["lexical", "semantic"]  # Optional: Control transcription matching (Marengo 3.0 only, requires "transcription" in search_options)
42
)
43
44
# 6. Process the search results
45
print("Search results:")
46
for clip in search_pager:
47
    print(
48
        f" video_id {clip.video_id} rank={clip.rank} start={clip.start} end={clip.end}"
49
    )

Step-by-step guide

Next steps

• Learn more about searching with text, image, and composed queries for best practices and advanced techniques.
• Explore entity search to find specific people in your videos.
• Learn query engineering techniques to refine your search queries.
• Use sorting to organize your search results.
• Apply grouping to cluster search results from the same video together.
• Implement filtering to narrow down results based on specific criteria.