This guide shows how you can create video embeddings using the Marengo video understanding model. For a list of available versions, complete specifications and input requirements for each version, see the Marengo page.
The Marengo video understanding model generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content.
For details on how your usage is measured and billed, see the Pricing page.
This section explains the key concepts and terminology used in this guide:
This guide shows how to upload your video as an asset and create embeddings asynchronously. You can also pass a URL or base64-encoded data inline instead of creating an asset; both are shown as commented-out lines in the code examples.
For videos under 10 minutes, synchronous processing returns embeddings immediately without polling. For details, see the Short videos (synchronous) section.
Customize your embeddings
You can configure embedding types (visual, audio, transcription), output format (separate, fused, or both), scope (clip or asset), and segmentation strategy (dynamic or fixed).
Use these embeddings for similarity search, content classification, clustering, recommendations, or Retrieval-Augmented Generation (RAG).
To use the platform, you need an API key:
Depending on the programming language you are using, install the TwelveLabs SDK by entering one of the following commands:
Your video files must meet the following requirements:
Copy and paste the code below, replacing the placeholders surrounded by <> with your values.
Create a client instance to interact with the TwelveLabs Video Understanding Platform.
Function call: You call the constructor of the TwelveLabs class.
Parameters:
api_key: The API key to authenticate your requests to the platform.Return value: An object of type TwelveLabs configured for making API calls.
Upload a video to create an asset.
Function call: You call the assets.create function.
Parameters:
method: The upload method for your asset. Use url for a publicly accessible or direct to upload a local file. This example uses url.url or file: The publicly accessible URL of your video or an opened file object in binary read mode. This example uses url.Return value: An object of type Asset. This object contains, among other information, a field named id representing the unique identifier of your asset.
For local files larger than 200 MB, use multipart uploads. Multipart uploads support automatic retry, progress tracking, parallel chunk uploads, and improved reliability, performance, and observability.
You only need this step for URL uploads larger than 200 MB. The platform processes these files asynchronously.
Function call: You call the assets.retrieve function.
Parameters:
asset_id: The unique identifier of your asset.Return value: An object of type Asset containing, among other information, a field named status representing the current status of the asset. Check this field until its value is ready.
Create an embedding task to start processing your video. This operation is asynchronous.
Function call: You call the embed.v_2.tasks.create function.
Parameters:
input_type: The type of content. Set this parameter to video.model_name: The model you want to use. This example uses marengo3.0.video: An object containing the following properties:
media_source: An object specifying the source of the video file. You can specify one of the following:
asset_id: The unique identifier of an asset from a previous upload.
url: The publicly accessible URL of the video file.
base_64_string: The base64-encoded video data.
This example uses the asset ID from the previous step.
(Optional) start_sec: The start time in seconds for processing the video file. By default, the platform processes videos from the beginning.
(Optional) end_sec: The end time in seconds for processing the video file. By default, the platform processes videos to the end of the video file.
(Optional) embedding_option: The types of embeddings to generate. Valid values are the following:
visual: Generates visual embeddings.audio: Generates embeddings for non-verbal audio (musical tones, beeping, environmental sounds).transcription: Generates embeddings for transcribed speech (the actual words spoken in the video).You can specify multiple values to generate different types of embeddings. The default value is ["visual", "audio", "transcription"].
(Optional) embedding_scope: The scope for which to generate embeddings. Valid values are the following:
clip: Generates one embedding for each segment.asset: Generates one embedding for the entire video file. Use this scope for videos up to 10-30 seconds to maintain optimal performance.You can specify multiple scopes to generate embeddings at different levels. The default value is ["clip", "asset"].
(Optional) segmentation: An object that specifies how the platform divides the video into segments. You can use one of the following strategies:
VideoSegmentation_Dynamic: Divides the video into segments that adapt to scene changes. Requires a property named dynamic with a min_duration_sec field specifying the minimum duration in seconds for each segment.VideoSegmentation_Fixed: Divides the video into segments of a fixed length. Requires a property named fixed with a duration_sec field specifying the exact duration in seconds for each segment.(Optional) embedding_type: An array specifying how to structure the embedding. Use this parameter only when embedding_option specifies two or more values. Valid values are the following:
separate_embedding: Returns separate embeddings for each modality specified in embedding_option.fused_embedding: Returns a single combined embedding that integrates all modalities into one vector.To receive both types in the same response, set this to ["separate_embedding", "fused_embedding"].
Return value: An object of type TasksCreateResponse containing, among other information, a field named id, which represents the unique identifier of your embedding task. You can use this identifier to track the status of your embedding task.
The platform requires some time to process videos. Poll the status of the embedding task until processing completes. This example uses a loop to check the status every 5 seconds.
Function call: You repeatedly call the embed.v_2.tasks.retrieve function until the task completes.
Parameters:
task_id: The unique identifier of your embedding task.Return value: An object of type EmbeddingTaskResponse containing, among other information, the following fields:
status: The current status of the task. The possible values are:
processing: The platform is creating the embeddings.ready: Processing is complete. Embeddings are available in the data field.failed: The task failed.data: When the status is ready, this field contains a list of embedding objects. Each embedding object includes:
embedding: The embedding vector (a list of floats).embedding_option: The type of embedding. Possible values are visual, audio, transcription, and fused. The platform returns fused only when embedding_type includes fused_embedding.embedding_scope: The scope of the embedding (clip or asset).start_sec: The start time of the segment in seconds.end_sec: The end time of the segment in seconds.For videos shorter than 10 minutes, you can use a synchronous approach that returns embeddings immediately without requiring polling.
All the fields of the video object function similarly to the asynchronous approach.