GuidesCreate embeddingsEmbed content at scale

Video embeddings

This guide shows how you can create video embeddings using the Marengo 3.5 video understanding model. For complete specifications and input requirements, see the Marengo 3.5 page.

The Marengo video understanding model generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content.

For details on how your usage is measured and billed, see the Pricing page.

Key concepts

This section explains the key concepts and terminology used in this guide:

  • Asset: Your uploaded content. Once created, you can reference the same asset across multiple operations without uploading the file again.
  • Embedding: Vector representation of your content.
  • Embedding task: An asynchronous operation for processing your content and creating embeddings. Contains a status and the resulting embeddings when complete.

Workflow

This guide shows how to create embeddings for a video file, one for each segment and one for the whole file, so your queries can match its content. The example uploads the video as an asset. You can also pass a URL or base64-encoded data inline instead of creating an asset; both are shown as commented-out lines in the code examples.

The platform processes your files asynchronously, one file per request. This example embeds one video; repeat the request for each file in your collection. To embed a query instead, see the Embed a query guide.

Customize your embeddings

You can configure the types of embeddings (visual and audio), the output format (separate, fused, or both), the scope (clip or asset), and the segmentation strategy (dynamic or fixed). You can also fold your own time-aligned text into the fused embedding and request a per-dimension uncertainty vector.

Use these embeddings for similarity search, content classification, clustering, recommendations, or Retrieval-Augmented Generation (RAG).

Retention policy

Embeddings created with the asynchronous method are stored for seven days. After this, you must recreate them to obtain the results again.

Prerequisites

  • To use the platform, you need an API key:

    1

    If you don’t have an account, sign up for a free account.

    2

    Go to the API Keys page.

    3

    If you need to create a new key, select the Create API Key button. Enter a name and set the expiration period. The default is 12 months.

    4

    Select the Copy icon next to your key to copy it to your clipboard.

  • Depending on the programming language you are using, install the TwelveLabs SDK by entering one of the following commands:

    $pip install --upgrade twelvelabs

  • Your video files must meet the following requirements:

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

1import time
2from twelvelabs import (
3    TwelveLabs,
4    AsyncVideoInputRequest,
5    MediaSource,
6    # For either segmentation strategy uncomment the next line:
7    # AsyncTemporalSegmentation,
8    # For dynamic segmentation uncomment the next two lines:
9    # TemporalSegmentation_Dynamic,
10    # TemporalSegmentationDynamicDynamic,
11    # For fixed segmentation uncomment the next two lines:
12    # TemporalSegmentation_Fixed,
13    # TemporalSegmentationFixedFixed,
14    # For time-based metadata uncomment the next line:
15    # TimeBasedMetadataEntry,
16)
17
18# 1. Initialize the client
19client = TwelveLabs(api_key="<YOUR_API_KEY>")
20
21# 2. Upload a video
22asset = client.assets.create(
23    method="url",
24    url="<YOUR_VIDEO_URL>" # Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported
25    # Or use method="direct" and file=open("<PATH_TO_VIDEO_FILE>", "rb") to upload a local file up to 200 MB
26)
27print(f"Created asset: id={asset.id}")
28
29# 3. Check the status of the asset
30print("Waiting for asset to be ready...")
31while True:
32    asset = client.assets.retrieve(asset.id)
33    if asset.status == "ready":
34        print("Asset is ready")
35        break
36    if asset.status == "failed":
37        raise RuntimeError(f"Asset processing failed: id={asset.id}")
38    time.sleep(5)
39
40# 4. Create video embeddings
41task = client.embed.v_2.tasks.create(
42    input_type="video",
43    model_name="marengo3.5",
44    video=AsyncVideoInputRequest(
45        media_source=MediaSource(
46            asset_id=asset.id,
47            # url="<YOUR_VIDEO_URL>", # Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported
48            # base_64_string="<BASE_64_ENCODED_DATA>",
49        ),
50        # start_sec=0,
51        # end_sec=10,
52        embedding_option=["visual", "audio"],
53        # embedding_scope=["clip", "asset"],
54        # embedding_type=["separate_embedding", "fused_embedding"],
55        # For dynamic segmentation:
56        # segmentation=AsyncTemporalSegmentation(
57        #     temporal=TemporalSegmentation_Dynamic(
58        #         dynamic=TemporalSegmentationDynamicDynamic(
59        #             min_duration_sec=3  # Minimum segment duration in seconds
60        #         )
61        #     )
62        # ),
63        # For fixed segmentation:
64        # segmentation=AsyncTemporalSegmentation(
65        #     temporal=TemporalSegmentation_Fixed(
66        #         fixed=TemporalSegmentationFixedFixed(
67        #             duration_sec=5  # Exact segment duration in seconds
68        #         )
69        #     )
70        # ),
71        # To fold your own time-aligned text into the fused embedding, uncomment the
72        # next lines and include "fused_embedding" in embedding_type:
73        # time_based_metadata=[
74        #     TimeBasedMetadataEntry(start=42.3, end=45.0, text="<YOUR_TEXT>"),
75        # ],
76    ),
77    # To receive per-dimension uncertainty, uncomment the next line and set
78    # embedding_scope above to a value that excludes "asset", such as ["clip"]:
79    # embedding_uncertainty=True,
80)
81print(f"Task ID: {task.id}")
82
83# 5. Monitor the status
84while True:
85    task = client.embed.v_2.tasks.retrieve(task_id=task.id)
86
87    if task.status == "ready":
88        print(f"Task completed")
89        break
90    elif task.status == "failed":
91        print("Task failed")
92        break
93    else:
94        print("Task still processing...")
95        time.sleep(5)
96
97# 6. Process the results
98print(f"\n{'='*80}")
99print(f"EMBEDDINGS SUMMARY: {len(task.data)} total embeddings")
100print(f"{'='*80}\n")
101
102for idx, embedding_data in enumerate(task.data, 1):
103    print(f"[{idx}/{len(task.data)}] {embedding_data.embedding_option.upper()} | {embedding_data.embedding_scope.upper()}")
104    print(f"├─ Time range: {embedding_data.start_sec}s - {embedding_data.end_sec}s")
105    print(f"├─ Dimensions: {len(embedding_data.embedding)}")
106    print(f"└─ First 10 values: {embedding_data.embedding[:10]}")
107    print()

Code explanation

1

Import the SDK and initialize the client

Create a client instance to interact with the TwelveLabs Video Understanding Platform.
Function call: You call the constructor of the TwelveLabs class.
Parameters:

  • api_key: The API key to authenticate your requests to the platform.

Return value: An object of type TwelveLabs configured for making API calls.

2

Upload a video

Upload a video to create an asset.
Function call: You call the assets.create function.
Parameters:

  • method: The upload method for your asset. Use url for a publicly accessible or direct to upload a local file. This example uses url.
  • url or file: The publicly accessible URL of your video or an opened file object in binary read mode. This example uses url.

Return value: An object of type Asset. This object contains, among other information, a field named id representing the unique identifier of your asset.

Note

For local files larger than 200 MB, use multipart uploads. Multipart uploads support automatic retry, progress tracking, parallel chunk uploads, and improved reliability, performance, and observability.

3

Check the status of the asset

Asset processing is asynchronous. Poll the status of the asset until it is ready before you use it.
Function call: You call the assets.retrieve function.
Parameters:

  • asset_id: The unique identifier of your asset.

Return value: An object of type Asset containing, among other information, a field named status representing the current status of the asset. Check this field until its value is ready.

4

Create an embedding task

Create an embedding task to start processing your video.
Function call: You call the embed.v_2.tasks.create function.
Parameters:

  • input_type: The type of content. Set this parameter to video.
  • model_name: The embedding model to use. This example uses marengo3.5.
  • (Optional) embedding_uncertainty: Set this parameter to true to receive a data[].embedding_uncertainty field in the response. This field is a per-dimension uncertainty vector with the same length as the embedding array. A higher value shows lower confidence in that dimension. To use this parameter, exclude the asset scope from the embedding_scope field. For example, set embedding_scope to ["clip"]. The field defaults to ["clip", "asset"], so a request that keeps the default returns a 400 error.
  • video: An object containing the following properties:

    • media_source: An object specifying the source of the video file. You can specify one of the following:

      • asset_id: The unique identifier of an asset from a previous upload.

      • url: The publicly accessible URL of the video file.

      • base_64_string: The base64-encoded video data.

        This example uses the identifier of the asset created in the previous step.

    • (Optional) start_sec: The start time in seconds for processing the video file. By default, the platform processes videos from the beginning.

    • (Optional) end_sec: The end time in seconds for processing the video file. By default, the platform processes videos to the end of the video file.

    • (Optional) embedding_option: The types of embeddings to generate. Valid values are the following:

      • visual: Generates embeddings based on visual content (scenes, objects, actions).
      • audio: Generates embeddings based on audio content (speech, music, and non-dialog audio).
      • transcription: Generates embeddings based on transcribed speech (the actual words spoken in the video). Legacy value that requires Marengo 3.0.

      You can specify multiple values to generate different types of embeddings. With Marengo 3.5 the default value is ["visual", "audio"].

    • (Optional) embedding_scope: The scope for which to generate embeddings. Valid values are the following:

      • clip: Generates one embedding for each segment.
      • asset: Generates one embedding for the entire video file. Use this scope for videos up to 10-30 seconds to maintain optimal performance.

      You can specify multiple scopes to generate embeddings at different levels. The default value is ["clip", "asset"].

    • (Optional) segmentation: An object that specifies how the platform divides the video into segments. Nest the strategy under the temporal field. You can use one of the following strategies:

      • temporal.dynamic: Divides the video into segments that adapt to scene changes. Requires a property named dynamic with a min_duration_sec field specifying the minimum duration in seconds for each segment.
      • temporal.fixed: Divides the video into segments of a fixed length. Requires a property named fixed with a duration_sec field specifying the exact duration in seconds for each segment.

    • (Optional) embedding_type: An array specifying how to structure the embedding. Use this parameter only when embedding_option specifies two or more values. Valid values are the following:

      • separate_embedding: Returns separate embeddings for each modality specified in embedding_option.
      • fused_embedding: Returns a single combined embedding that integrates all modalities into one vector. With Marengo 3.5, this value requires the time_based_metadata field.

      To receive both types in the same response, set this to ["separate_embedding", "fused_embedding"].

    • (Optional) time_based_metadata: An array of your own time-aligned text entries, such as a stats feed or scene descriptions. The platform folds each entry into the fused embedding of the segments it overlaps in time, and it affects only that embedding. This parameter requires the fused_embedding value in the embedding_type parameter. Each entry contains the following properties:

      • start: The start time of the entry in seconds, measured from the beginning of the asset.
      • end: The end time of the entry in seconds. Set the same value as start for an event that happens at a single point in time.
      • text: The text to fold into the fused embedding of the overlapping segments.

Return value: An object of type TasksCreateResponse containing, among other information, a field named id, which represents the unique identifier of your embedding task. You can use this identifier to track the status of your embedding task.

5

Monitor the status

The platform requires some time to process videos. Poll the status of the embedding task until it is ready. This example uses a loop to check the status every 5 seconds.
Function call: You repeatedly call the embed.v_2.tasks.retrieve function until the task completes.

Parameters:

  • task_id: The unique identifier of your embedding task.

Return value: An object of type EmbeddingTaskResponse containing, among other information, the following fields:

  • status: The current status of the task. The possible values are:
    • processing: The platform is creating the embeddings.
    • ready: Processing is complete. Embeddings are available in the data field.
    • failed: The task failed.
  • data: When the status is ready, this field contains a list of embedding objects. Each embedding object includes:
    • embedding: The embedding vector (a list of floats).
    • embedding_uncertainty: A per-dimension uncertainty vector with the same length as the embedding array. A higher value shows lower confidence in that dimension. Present when the request sets embedding_uncertainty to true.
    • embedding_option: The type of embedding. Possible values are visual, audio, transcription, and fused. The platform returns fused only when embedding_type includes fused_embedding.
    • embedding_scope: The scope of the embedding (clip or asset).
    • start_sec: The start time of the segment in seconds.
    • end_sec: The end time of the segment in seconds.
6

Process the results

This example iterates through the embeddings in the data field and prints the embedding type, scope, time range, dimensions, and the first 10 vector values for each segment.