This guide shows how you can create document embeddings using the Marengo 3.5 video understanding model. For complete specifications and input requirements, see the Marengo 3.5 page.

The Marengo video understanding model generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content.

For details on how your usage is measured and billed, see the Pricing page.

Key concepts

This section explains the key concepts and terminology used in this guide:

Asset: Your uploaded content. Once created, you can reference the same asset across multiple operations without uploading the file again.
Embedding: Vector representation of your content.
Embedding task: An asynchronous operation for processing your content and creating embeddings. Contains a status and the resulting embeddings when complete.

Workflow

This guide shows how to create embeddings for documents (PDF, plain text, and Markdown files), so your queries can match their content. The example uploads the PDF file as an asset. You can also pass a URL or base64-encoded data inline instead of creating an asset; both are shown as commented-out lines in the code examples.

The platform processes your files asynchronously, one file per request. This example embeds one document; repeat the request for each file in your collection. To embed a query instead, see the Embed a query guide.

You can configure what the platform embeds. For a PDF file, you can create one embedding per rendered page, one for the entire file, one for its extracted text, or split each rendered page into quadrants and receive five embeddings per page. For a plain text or Markdown file, you can create one embedding for the entire file or one for each chunk of whole sentences.

Use these embeddings for similarity search, content classification, clustering, recommendations, or Retrieval-Augmented Generation (RAG).

Retention policy

Embeddings created with the asynchronous method are stored for seven days. After this, you must recreate them to obtain the results again.

Prerequisites

To use the platform, you need an API key:

1
If you don’t have an account, sign up for a free account.
2
Go to the API Keys page.
3
If you need to create a new key, select the Create API Key button. Enter a name and set the expiration period. The default is 12 months.
4
Select the Copy icon next to your key to copy it to your clipboard.
Depending on the programming language you are using, install the TwelveLabs SDK by entering one of the following commands:
```
pip install --upgrade twelvelabs
```
Your documents must meet the following requirements:
- Upload limits: Local documents up to 200 MB, or public document URLs up to 512 MB.
- Formats: PDF (.pdf), plain text (.txt), and Markdown (.md).
- Pages: A PDF file has a page allowance of 64 pages for each MB of file size. A 0.5 MB file is allowed 64 pages, and a 4 MB file is allowed 256 pages. The quadrants strategy counts each page five times against this allowance. Plain text and Markdown files have no page allowance.
- Model capabilities: See the complete input requirements for Marengo 3.5.

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

import time
from twelvelabs import (
    TwelveLabs,
    AsyncDocumentInputRequest,
    MediaSource,
    # To segment a document uncomment the next line:
    # DocumentSegmentation,
    # For a PDF file (spatial segmentation) uncomment the next line:
    # DocumentSpatialSegmentation,
    # For a plain text or Markdown file (sequential segmentation) uncomment the next line:
    # DocumentSequentialSegmentation,
)
# 1. Initialize the client
client = TwelveLabs(api_key="<YOUR_API_KEY>")
# 2. Upload a document
asset = client.assets.create(
    method="url",
    url="<YOUR_DOCUMENT_URL>" # Use direct links to raw files. Cloud storage sharing links are not supported
    # Or use method="direct" and file=open("<PATH_TO_DOCUMENT_FILE>", "rb") to upload a local file up to 200 MB
)
print(f"Created asset: id={asset.id}")
# 3. Check the status of the asset
print("Waiting for asset to be ready...")
while True:
    asset = client.assets.retrieve(asset.id)
    if asset.status == "ready":
        print("Asset is ready")
        break
    if asset.status == "failed":
        raise RuntimeError(f"Asset processing failed: id={asset.id}")
    time.sleep(5)
# 4. Create an embedding task
task = client.embed.v_2.tasks.create(
    input_type="document",
    model_name="marengo3.5",
    document=AsyncDocumentInputRequest(
        media_source=MediaSource(
            asset_id=asset.id,
            # url="<YOUR_DOCUMENT_URL>", # Use direct links to raw files. Cloud storage sharing links are not supported
            # base_64_string="<BASE_64_ENCODED_DATA>",
        ),
            # segmentation=DocumentSegmentation(
            #     spatial=DocumentSpatialSegmentation(strategy="quadrants"), # Split each page into quadrants: five embeddings per page, at five times the token cost
            # ),
            # For a plain text or Markdown file, chunk the text into whole sentences instead:
            # segmentation=DocumentSegmentation(
            #     sequential=DocumentSequentialSegmentation(strategy="sentence", max_sentences=5), # strategy and max_sentences are required
            #     # overlap_sentences=1, # Optional: repeat the last sentences of each chunk at the start of the next
            # ),
            # embedding_option=["visual", "text"], embedding_scope=["asset"], # Whole-file visual + whole-file text
            # embedding_type=["separate_embedding"], # The only valid value for documents
            # embedding_scope=["local"], # One embedding per page
            # embedding_scope=["local", "asset"], # Per-page embeddings and one whole-file embedding
        ),
    # embedding_uncertainty=True,
    # embedding_dimension=512,
)
print(f"Task ID: {task.id}")
# 5. Monitor the status
while True:
    task = client.embed.v_2.tasks.retrieve(task_id=task.id)
    if task.status == "ready":
        print("Task completed")
        break
    elif task.status == "failed":
        print("Task failed")
        break
    else:
        print("Task still processing...")
        time.sleep(5)
# 6. Process the results
print(f"Number of embeddings: {len(task.data)}")
if task.metadata is not None and task.metadata.embedding_dimension is not None:
    print(f"Embedding dimensions (metadata.embedding_dimension): {task.metadata.embedding_dimension}")
for embedding_data in task.data:
    print(f"[{embedding_data.embedding_scope}]")
    if embedding_data.start_page_number is not None:
        print(f"Pages: {embedding_data.start_page_number}-{embedding_data.end_page_number}")
    if embedding_data.quadrant is not None:
        print(f"Quadrant: {embedding_data.quadrant}")
    if embedding_data.chunk_index is not None:
        print(f"Chunk index: {embedding_data.chunk_index}")
    if embedding_data.embedding_uncertainty is not None:
        print(f"First 10 uncertainty values: {embedding_data.embedding_uncertainty[:10]}")
    print(f"Embedding dimensions: {len(embedding_data.embedding)}")
    print(f"First 10 values: {embedding_data.embedding[:10]}")

Code explanation

Python

Node.js

Import the SDK and initialize the client

Create a client instance to interact with the TwelveLabs Video Understanding Platform.
Function call: You call the constructor of the TwelveLabs class.
Parameters:

api_key: The API key to authenticate your requests to the platform.

Return value: An object of type TwelveLabs configured for making API calls.

Upload a document

Upload a document to create an asset.
Function call: You call the assets.create function.
Parameters:

method: The upload method for your asset. Use url for a publicly accessible file or direct to upload a local file. This example uses url.
url or file: The publicly accessible URL of your document file or an opened file object in binary read mode. This example uses url.

Return value: An object of type Asset. This object contains, among other information, a field named id representing the unique identifier of your asset.

Check the status of the asset

Asset processing is asynchronous. Poll the status of the asset until it is ready before you use it.
Function call: You call the assets.retrieve function.
Parameters:

asset_id: The unique identifier of your asset.

Return value: An object of type Asset containing, among other information, a field named status representing the current status of the asset. Check this field until its value is ready.

Create an embedding task

Create an embedding task to start processing your document.
Function call: You call the embed.v_2.tasks.create function.
Parameters:

input_type: The type of content. Set this parameter to document.
model_name: The embedding model to use. This example uses marengo3.5.
(Optional) embedding_uncertainty: Set this parameter to true to receive a data[].embedding_uncertainty field in the response. This field is a per-dimension uncertainty vector with the same length as the embedding array. A higher value shows lower confidence in that dimension.
(Optional) embedding_dimension: The number of dimensions of the embedding: 128, 256, or 512. The default is 512. Applies to the whole task. To embed the same content at a different length, create a second task.
document: An object containing the following properties:
- media_source: An object specifying the source of the document file. Specify one of the following:
  - asset_id: The unique identifier of an asset from a previous upload.
  - url: The publicly accessible URL of the document file.
  - base_64_string: The base64-encoded document data.
    
    This example uses the identifier of the asset created in the previous step.
- (Optional) segmentation: Controls how the platform divides your document before embedding it. The object has two fields: set spatial for a PDF file, or sequential for a plain text or Markdown file. If you provide neither field, the platform returns a 400 error.
  - spatial: For a PDF file, sets how the platform divides each rendered page. Set the strategy field to the standard value (the default) for one embedding per page. Set it to the quadrants value to split each page on a 2×2 grid and receive five embeddings per page: one for the whole page and one per quarter. The quadrants strategy uses five times as many tokens as the standard strategy and counts each page five times against the page allowance of the file. Requires embedding_option: ["visual"] and embedding_scope: ["local"].
  - sequential: For a plain text or Markdown file, divides the text into chunks of whole sentences. Set strategy to sentence and max_sentences to the maximum number of sentences per chunk. Both fields are required, and max_sentences has no default. A chunk that exceeds the context window of the model fails the task, so choose a value small enough that every chunk fits. Optionally, set overlap_sentences to repeat the last sentences of each chunk at the start of the next; it must be less than max_sentences. Requires embedding_option: ["text"] and embedding_scope: ["local"].
- (Optional) embedding_option: The types of embeddings to generate for the document. Valid values are the following:
  - visual: Embeds the rendered pages. Valid for PDF files.
  - text: Generates embeddings from the text content. The default for plain text and Markdown files. For a PDF file, valid only with the asset scope. You can request multiple types in one task: for example, embedding_option: ["visual", "text"] with embedding_scope: ["asset"] on a PDF file returns one whole-file visual embedding and one whole-file text embedding together.
- (Optional) embedding_scope: The scope for which to generate embeddings. Valid values are the following:
  - local: Returns one embedding for each part of the file. For a PDF file, each part is a rendered page, and this is the default. For a plain text or Markdown file, each part is a chunk of whole sentences, and the segmentation.sequential field is required.
  - asset: Returns one embedding for the entire file. The default for plain text and Markdown files. You can request multiple scopes in one task: for example, embedding_scope: ["local", "asset"] on a PDF file returns the per-page embeddings and the whole-file embedding together.
- (Optional) embedding_type: Specifies how to structure the embedding. The only valid value is separate_embedding. Documents have a single modality, so the platform returns a 400 error if you set fused_embedding.

Text embeddings of PDF files

To embed the extracted text of a PDF file, set embedding_option to ["text"] and embedding_scope to ["asset"]. If you omit embedding_scope, the platform uses the PDF default (local). The local scope supports only the visual option, so the request fails with a 400 error.

Return value: An object of type TasksCreateResponse containing, among other information, a field named id, which represents the unique identifier of your embedding task. You can use this identifier to track the status of your embedding task. The object also includes a metadata.embedding_dimension field, which contains the number of dimensions of the embeddings this task returns.

Monitor the status

The platform requires some time to process documents. Poll the status of the embedding task until it is ready. This example uses a loop to check the status every 5 seconds.
Function call: You repeatedly call the embed.v_2.tasks.retrieve function until the task completes.

Parameters:

task_id: The unique identifier of your embedding task.

Return value: An object of type EmbeddingTaskResponse containing, among other information, the following fields:

status: The current status of the task. The possible values are:
- processing: The platform is creating the embeddings.
- ready: Processing is complete. Embeddings are available in the data field.
- failed: The task failed.
data: When the status is ready, this field contains a list of embedding objects. Each embedding object includes:
- embedding: The embedding vector (a list of floats).
- embedding_uncertainty: A per-dimension uncertainty vector with the same length as the embedding array. A higher value shows lower confidence in that dimension. Present when the request sets embedding_uncertainty to true.
- embedding_scope: The scope of the embedding. For a PDF file, a request that sets the local scope returns page. For a plain text or Markdown file, a request that sets the local scope returns chunk. A request that sets the asset scope returns asset.
- quadrant: The quarter of the page this embedding covers, such as top_left. Present only when the segmentation.spatial.strategy is quadrants; null on the whole-page embedding. All five embeddings of a page share the same page numbers and page scope, so use this field to tell them apart.
- chunk_index: The position of the chunk in the file, counting from 0. Present only when the segmentation.sequential field is set. The data array has no ordering guarantee, so read this field to reconstruct chunk order.
- start_page_number: The first page this embedding covers, counting from 1. The platform returns this field only for page-level embeddings of a PDF file.
- end_page_number: The last page this embedding covers, counting from 1 and including that page.
metadata.embedding_dimension: The number of dimensions of the embeddings this task returns. Only Marengo 3.5.

Process the results

This example iterates through the embeddings in the data field and prints the scope, dimensions, and first 10 vector values for each embedding, plus the conditional fields from the previous step when present.