This guide shows how to migrate your applications to the 1.3 version of the API, which introduces significant improvements to the video understanding capabilities of the platform, simplified modalities, and a streamlined endpoint structure.

📘

Note

All changes described in this guide apply to the 1.3 version of the API. Features marked as deprecated, renamed, or removed are unavailable in v1.3 but remain functional in v1.2 until further notice. The only exception is the Classify API, which will be deprecated in v1.2 on Feb 28, 2025.

What's new in v1.3?

• Marengo 2.7: A new version of the Marengo video understanding model has been released. The 1.3 version of the API version only supports Marengo 2.7. This new version improves accuracy and performance in the following areas:
  • Multimodal processing that combines visual, audio, and text elements.
  • Fine-grained image-to-video search: detect brand logos, text, and small objects (as small as 10% of the video frame).
  • Improvement in motion search capability.
  • Counting capabilities.
  • More nuanced audio comprehension: music, lyrics, sound, and silence.
• Simplified modalities:
  • visual: includes objects, actions, text OCR, logos.
  • audio: includes speech, music, and ambient sounds.
  • conversation has been deprecated.
  • text_in_video and logo are now part of visual.
• Streamlined endpoint structure: Several endpoints and parameters have been deprecated, removed, or renamed.

📘

Note

This guide presents the changes to the API. Since the SDKs reflect the structure of the API, review the Migration examples section below and the relevant SDK reference sections to understand how these changes have been implemented:

• Python SDK
• Node.js SDK

Breaking changes

This section presents the changes that require updates to your code and includes the following subsections:

• Global changes that affect multiple endpoints
• Changes organized by endpoint and functionality (example: upload videos, manage indexes, etc.)

In the sections below, see the Required Action column for each change, then use the corresponding example in the Migration examples section to update your code.

Global changes

Deprecated endpoints

Upload videos

Manage indexes

Manage videos

Search

Generate text from video

Non-breaking changes

These changes add new functionality while maintaining backward compatibility.

Upload videos

Migration steps

Migrating to v1.3 involves two main steps:

1. Update your integration
2. Use the migration examples to update your code.

1. Update your integration

Choose the appropriate method based on how you interact with the Twelve Labs API:

• Official SDKs: Install the latest version that supports v1.3.
• HTTP client: Update your base URL.

pip3 install twelvelabs --upgrade

 npm install twelvelabs-js@latest

API_URL="https://api.twelvelabs.io/v1.3"

2. Migration examples

Below are examples showing how to update your code for key breaking changes. Choose the examples matching your integration type.

Create indexes

Creating an index in version 1.3 includes the following key changes:

• Renamed parameters: The parameters that previously began with engine* have now been renamed to model*.
• Simplified modalities: The previous modalities of [visual, conversation, text_in_video, logo] have been simplified to [visual, audio].
• Marengo version update: Use "marengo2.7" instead of "marengo2.6".

models = [
        {
          "name": "marengo2.7",
          "options": ["visual", "audio"]
        },
        {
            "name": "pegasus1.1",
            "options": ["visual", "audio"]
        }
    ]
index = client.index.create(
    name="<YOUR_INDEX_NAME>",
    models=models,
    addons=["thumbnail"] # Optional
)

const models = [
  {
    name: 'marengo2.7',
    options: ['visual', 'audio'],
  },
  {
    name: 'pegasus1.1',
    options: ['visual', 'audio'],
  },
];

let index = await client.index.create({
  name: '<YOUR_INDEX_NAME>',
  models: models,
  addons: ['thumbnail'], // Optional
});

curl --request POST \
  --url https://api.twelvelabs.xyz/v1.3/indexes \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: <YOUR_API_KEY>' \
  --data '{
  "models": [
    {
      "model_name": "marengo2.7",
      "model_options": ["visual", "audio"]
    },
    {
      "model_name": "pegasus1.1",
      "model_options": ["visual", "audio"]
    }
  ],
  "addons": ["thumbnail"],
  "index_name":"<YOUR_INDEX_NAME>"
}'

Perform a search request

Performing a search request includes the following key changes:

• Simplified modalities: The previous modalities of [visual, conversation, text_in_video, logo] have been simplified to [visual, audio].
• Deprecated parameter: The conversation_option parameter has been deprecated.
• Streamlined response: The metadata and modules fields in the response have been deprecated.

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>", 
  query_text="<YOUR_QUERY>", 
  options=["visual", "audio"]
)

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>'
  queryText: '<YOUR_QUERY>',
  options: ['visual'],
});

curl --request POST \
  --url https://api.twelvelabs.xyz/v1.3/search \
  --header 'Content-Type: multipart/form-data' \
  --header 'x-api-key: <YOUR_API_KEY>' \
  --form index_id=<YOUR_INDEX_ID> \
  --form search_options=visual \
  --form search_options=conversation \
  --form 'query_text=<YOUR_QUERY>'

Create embeddings

Creating embeddings includes the following key changes:

• Marengo version update: Use "Marengo-retrieval-2.7" instead of "Marengo-retrieval-2.6".
• Renamed parameter: The parameters that previously began with engine* have now been renamed to model*.

The following example creates a text embedding, but the principles demonstrated are similar for image, audio, and video embeddings:

res = client.embed.create(
  model_name="Marengo-retrieval-2.7",
  text="<YOUR_TEXT>",
)

let res = await client.embed.create({
  modelName: "Marengo-retrieval-2.7",
  text: "<YOUR_TEXT>",
});

curl --request POST \
  --url https://api-dev.twelvelabs.xyz/v1.3/embed \
  --header 'Content-Type: multipart/form-data' \
  --header 'x-api-key: <YOUR_API_KEY' \
  --form model_name=Marengo-retrieval-2.7 \
  --form 'text=<YOUR_TEXT>t'

Use Pegasus to classify videos

The Pegasus video understanding model offers flexible video classification through its text generation capabilities. You can use established category systems like YouTube video categories or IAB Tech Lab Content Taxonomy . You can also define custom categories for your specific needs.

The example below classifies a video based on YouTube's video categories:

res = client.generate.text(
  video_id="<YOUR_VIDEO_ID>",
  prompt="Classify this video using up to five labels from YouTube standard content categories. Provide the results in the JSON format."
)

const text = await client.generate.text(
  '<YOUR_VIDEO_ID>',
  'Classify this video using up to five labels from YouTube standard content categories. Provide the results in the JSON format.',
);

curl --request POST \
  --url https://api.twelvelabs.xyz/v1.2/generate \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: <YOUR_API_KEY>' \
  --data '{
  "video_id": "<YOUR_VIDEO_ID>",
  "prompt": "Classify this video using up to five labels from YouTube standard content categories. Provide the results in the JSON format."
}'

Detect logos

You can search for logos using text or image queries:

• Text queries: For logos that include text (example: Nike)
• Image queries: For logos without text (example: Apple's apple symbol).

The following example searches for the Nike logo using a text query:

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>", 
  query_text="Nike", 
  options=["visual"]
)

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>'
  queryText: 'Nike',
  options: ['visual'],
});

curl --request POST \
  --url https://api.twelvelabs.xyz/v1.3/search \
  --header 'Content-Type: multipart/form-data' \
  --header 'x-api-key: <YOUR_API_KEY>' \
  --form index_id=<YOUR_INDEX_ID> \
  --form search_options=visual \
  --form search_options=conversation \
  --form 'query_text=Nike'

The following example searches for the Apple logo using an image query:

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query_media_type="image",
    query_media_url="https://logodownload.org/wp-content/uploads/2013/12/apple-logo-16.png,
    options=["visual"]
)

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  queryMediaType: 'image',
  queryMediaUrl: 'https://logodownload.org/wp-content/uploads/2013/12/apple-logo-16.png',
  options: ['visual']
});

curl --request POST \
  --url https://api.twelvelabs.io/v1.3/search \
  --header 'Content-Type: multipart/form-data' \
  --header 'x-api-key: <YOUR_API_KEY>' \
  --form index_id=<YOUR_INDEX_ID> \
  --form search_options=visual \
  --form query_media_type=image \
  --form query_media_url=https://logodownload.org/wp-content/uploads/2013/12/apple-logo-16.png

Search for text shown in videos

To search for text in videos, use text queries that target either on-screen text or spoken words in transcriptions rather than objects or concepts. The platform searches across both:

• Text shown on screen (such as titles, captions, or signs)
• Spoken words from audio transcriptions

Note that the platform may return both textual and visual matches. For example, searching for the word "smartphone" might return:

• Segments where "smartphone" appears as on-screen text.
• Segments where "smartphone" is spoken.
• Segments where smartphones are visible as objects.

The example below finds all the segments where the word "innovation" appears as on-screen text or as a spoken word in transcriptions:

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>", 
  query_text="Innovation", 
  options=["visual"]
)

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>'
  queryText: 'Innovation',
  options: ['visual'],
});

curl --request POST \
  --url https://api.twelvelabs.xyz/v1.2/generate \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: <YOUR_API_KEY>' \
  --data '{
  "video_id": "<YOUR_VIDEO_ID>",
  "prompt": "Innovation"
}'

Additional resources

• API Rererence
• Python SDK Reference
• Node.js SDK Reference
• Discord