Use the TwelveLabs Video Understanding Platform to find specific moments in your video content using natural language queries or reference images. The platform analyzes videos by integrating images, audio, speech, and text, offering a deeper understanding than single-modal methods. It captures complex relationships between these elements, detects subtle details, and supports natural language queries and images for intuitive and precise use.

Key features:

• Improved accuracy: Multimodal integration enhances accuracy.
• Easy interaction: Natural language queries simplify searches.
• Advanced search: Enables image-based queries for precise results.
• Fewer errors: Multi-faceted analysis reduces misinterpretation.
• Time savings: Quickly finds relevant clips without manual review.

Use cases:

• Spoken word search: Find video segments where specific words or phrases are spoken.
• Visual element search: Locate video segments that match descriptions of visual elements or scenes.
• Action or event search: Identify video segments that depict specific actions or events.
• Image similarity search: Find video segments that visually resemble a provided image.
• Entity search: Locate video segments containing specific people, car models, animal species, or branded objects with improved accuracy.

For details on how your usage is measured and billed, see the Pricing page.

Key concepts

This section explains the key concepts and terminology used in this guide:

• Index: A container that organizes your video content
• Asset: Your uploaded content. Once created, you can reference the same asset across multiple operations without uploading the file again.
• Indexed asset: An asset that has been indexed and is ready for downstream tasks.

Workflow

Upload and index your videos before you search them. The platform indexes videos asynchronously. You can search your videos after indexing completes. Search results show video segments that match your search terms.

Types of search queries

The platform supports three types of search queries:

• Text queries: Search using natural language descriptions of visual elements, actions, sounds, or spoken words
• Image queries: Search using images to find visually similar content in your videos
• Composed queries: Combine text descriptions with images for more precise results

For guidance on choosing the correct query type, see the Search with text, image, and composed queries page.

Search scope

You can search within a single index per request. You cannot search at the video level or across multiple indexes simultaneously.

Customize your search

You can customize your search in the following ways:

• Specify which modalities to use: visual, audio, or transcription (spoken words)
• Choose how to combine modalities: use the or or and operators
• For searches within spoken words, select the match type: lexical, semantic, or both

Prerequisites

• To use the platform, you need an API key:

  1

  If you don’t have an account, sign up for a free account.

  2

  Go to the API Keys page.

  3

  If you need to create a new key, select the Create API Key button. Enter a name and set the expiration period. The default is 12 months.

  4

  Select the Copy icon next to your key to copy it to your clipboard.

• Depending on the programming language you are using, install the TwelveLabs SDK by entering one of the following commands:

  PythonNode.js

  $
  pip install twelvelabs

• Your video files must meet the following requirements:

  • Upload limits: Public video URLs up to 4 GB or local video files up to 200 MB. For local files up to 4 GB, see the Upload and processing methods page.
  • Model capabilities: See the complete requirements for resolution, aspect ratio, and supported formats.

• If you wish to use images as queries, ensure that your image file meet the requirements.

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

1
import time
2
from twelvelabs import TwelveLabs
3
4
# 1. Initialize the client
5
client = TwelveLabs(api_key="<YOUR_API_KEY>")
6
7
# 2. Create an index
8
# An index is a container for organizing your video content
9
index = client.indexes.create(
10
    index_name="<YOUR_INDEX_NAME>",
11
    models=[{"model_name": "marengo3.0", "model_options": ["visual", "audio"]}]
12
)
13
if not index.id:
14
    raise RuntimeError("Failed to create an index.")
15
print(f"Created index: id={index.id}")
16
17
# 3. Upload a video
18
asset = client.assets.create(
19
    method="url",
20
    url="<YOUR_VIDEO_URL>" # Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported
21
    # Or use method="direct" and file=open("<PATH_TO_VIDEO_FILE>", "rb") to upload a local file up to 200 MB
22
)
23
print(f"Created asset: id={asset.id}")
24
25
# 4. Check the status of the asset
26
print("Waiting for asset to be ready...")
27
while True:
28
    asset = client.assets.retrieve(asset.id)
29
    if asset.status == "ready":
30
        print("Asset is ready")
31
        break
32
    if asset.status == "failed":
33
        raise RuntimeError(f"Asset processing failed: id={asset.id}")
34
    time.sleep(5)
35
36
# 5. Index your video
37
indexed_asset = client.indexes.indexed_assets.create(
38
    index_id=index.id,
39
    asset_id=asset.id,
40
    # enable_video_stream=True
41
)
42
print(f"Created indexed asset: id={indexed_asset.id}")
43
44
# 6. Monitor the indexing process
45
print("Waiting for indexing to complete.")
46
while True:
47
    indexed_asset = client.indexes.indexed_assets.retrieve(
48
        index_id=index.id,
49
        indexed_asset_id=indexed_asset.id
50
    )
51
    print(f"  Status={indexed_asset.status}")
52
53
    if indexed_asset.status == "ready":
54
        print("Indexing complete!")
55
        break
56
    elif indexed_asset.status == "failed":
57
        raise RuntimeError("Indexing failed")
58
59
    time.sleep(5)
60
61
# 7. Perform a search request
62
search_results = client.search.query(
63
    index_id=index.id,
64
    query_text="<YOUR_QUERY>",
65
    search_options=["visual", "audio"]
66
    # operator="or" # Optional: Use "and" to find segments matching all modalities
67
    # transcription_options=["lexical", "semantic"]  # Optional: Control transcription matching, requires "transcription" in search_options)
68
)
69
70
# 8. Process the search results
71
print("\nSearch results:")
72
print("Each result shows a video clip that matches your query:\n")
73
for i, clip in enumerate(search_results):
74
    print(f"Result {i + 1}:")
75
    print(f"  Video ID: {clip.video_id}")  # Unique identifier of the video
76
    print(f"  Rank: {clip.rank}")  # Relevance ranking (1 = most relevant)
77
    print(f"  Time: {clip.start}s - {clip.end}s", end="\n\n")  # When this moment occurs in the video

Code explanation

Next steps

• Learn more about searching with text, image, and composed queries for best practices and advanced techniques.
• Explore entity search to find specific people in your videos.
• Learn query engineering techniques to refine your search queries.
• Apply grouping to cluster search results from the same video together.
• Implement filtering to narrow down results based on specific criteria.