Use the TwelveLabs Video Understanding Platform to find specific moments in your video content using natural language queries or reference images. The platform analyzes videos by integrating images, audio, speech, and text, offering a deeper understanding than single-modal methods. It captures complex relationships between these elements, detects subtle details, and supports natural language queries and images for intuitive and precise use.

Key features:

• Improved accuracy: Multimodal integration enhances accuracy.
• Easy interaction: Natural language queries simplify searches.
• Advanced search: Enables image-based queries for precise results.
• Fewer errors: Multi-faceted analysis reduces misinterpretation.
• Time savings: Quickly finds relevant clips without manual review.

Use cases:

• Spoken word search: Find video segments where specific words or phrases are spoken.
• Visual element search: Locate video segments that match descriptions of visual elements or scenes.
• Action or event search: Identify video segments that depict specific actions or events.
• Image similarity search: Find video segments that visually resemble a provided image.
• Entity search: Locate video segments containing specific people, car models, animal species, or branded objects with improved accuracy (Marengo 3.0 only).

For details on how your usage is measured and billed, see the Pricing page.

Key concepts

This section explains the key concepts and terminology used in this guide:

• Index: A container that organizes your video content
• Asset: Your uploaded file
• Indexed asset: A video that has been indexed and is ready for downstream tasks

Workflow

Upload and index your videos before you search them. The platform indexes videos asynchronously. You can search your videos after indexing completes. Search results show video segments that match your search terms.

Types of search queries

The platform supports three types of search queries:

• Text queries: Search using natural language descriptions of visual elements, actions, sounds, or spoken words
• Image queries: Search using images to find visually similar content in your videos
• Composed queries: Combine text descriptions with images for more precise results (Marengo 3.0 only)

For guidance on choosing the correct query type, see the Search with text, image, and composed queries page.

Search scope

You can search within a single index per request. You cannot search at the video level or across multiple indexes simultaneously.

Customize your search

You can customize your search in the following ways:

• Specify which modalities to use: visual, audio, or transcription (spoken words)
• Choose how to combine modalities: use the or or and operators
• For searches within spoken words, select the match type: lexical, semantic, or both

Prerequisites

• To use the platform, you need an API key:

  1

  If you don’t have an account, sign up for a free account.

  2

  Go to the API Keys page.

  3

  Select the Copy icon next to your key.

• Depending on the programming language you are using, install the TwelveLabs SDK by entering one of the following commands:

  PythonNode.js

  $
  pip install twelvelabs

• Your video files must meet the following requirements:

  • For this guide: Files up to 4 GB when using publicly accessible URLs or 200 MB for local files
  • Model capabilities: See the complete requirements for resolution, aspect ratio, and supported formats.

  For other upload methods with different limits, see the Upload methods page.

• If you wish to use images as queries, ensure that your image file meet the requirements.

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

1
import time
2
from twelvelabs import TwelveLabs
3
4
# 1. Initialize the client
5
client = TwelveLabs(api_key="<YOUR_API_KEY>")
6
7
# 2. Create an index
8
# An index is a container for organizing your video content
9
index = client.indexes.create(
10
    index_name="<YOUR_INDEX_NAME>",
11
    models=[{"model_name": "marengo3.0", "model_options": ["visual", "audio"]}]
12
)
13
if not index.id:
14
    raise RuntimeError("Failed to create an index.")
15
print(f"Created index: id={index.id}")
16
17
# 3. Upload a video
18
asset = client.assets.create(
19
    method="url",
20
    url="<YOUR_VIDEO_URL>" # Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported
21
    # Or use method="direct" and file=open("<PATH_TO_VIDEO_FILE>", "rb") to upload a file from the local file system
22
)
23
print(f"Created asset: id={asset.id}")
24
25
# 4. Index your video
26
indexed_asset = client.indexes.indexed_assets.create(
27
    index_id=index.id,
28
    asset_id=asset.id,
29
    # enable_video_stream=True
30
)
31
print(f"Created indexed asset: id={indexed_asset.id}")
32
33
# 5. Monitor the indexing process
34
print("Waiting for indexing to complete.")
35
while True:
36
    indexed_asset = client.indexes.indexed_assets.retrieve(
37
        index_id=index.id,
38
        indexed_asset_id=indexed_asset.id
39
    )
40
    print(f"  Status={indexed_asset.status}")
41
42
    if indexed_asset.status == "ready":
43
        print("Indexing complete!")
44
        break
45
    elif indexed_asset.status == "failed":
46
        raise RuntimeError("Indexing failed")
47
48
    time.sleep(5)
49
50
# 6. Perform a search request
51
search_results = client.search.query(
52
    index_id=index.id,
53
    query_text="<YOUR_QUERY>",
54
    search_options=["visual", "audio"]
55
    # operator="or" # Optional: Use "and" to find segments matching all modalities
56
    # transcription_options=["lexical", "semantic"]  # Optional: Control transcription matching (Marengo 3.0 only, requires "transcription" in search_options)
57
)
58
59
# 7. Process the search results
60
print("\nSearch results:")
61
print("Each result shows a video clip that matches your query:\n")
62
for i, clip in enumerate(search_results):
63
    print(f"Result {i + 1}:")
64
    print(f"  Video ID: {clip.video_id}")  # Unique identifier of the video
65
    print(f"  Rank: {clip.rank}")  # Relevance ranking (1 = most relevant)
66
    print(f"  Time: {clip.start}s - {clip.end}s", end="\n\n")  # When this moment occurs in the video

Code explanation

Next steps

• Learn more about searching with text, image, and composed queries for best practices and advanced techniques.
• Explore entity search to find specific people in your videos.
• Learn query engineering techniques to refine your search queries.
• Use sorting to organize your search results.
• Apply grouping to cluster search results from the same video together.
• Implement filtering to narrow down results based on specific criteria.