Upgrade your SDK

$
pip install --upgrade twelvelabs

Update your Python input classes

Skip this step if you use the Node.js SDK, or if you do not type-check your Python code.

In the Python SDK, the AsyncVideoInputRequest and AsyncAudioInputRequest classes replace VideoInputRequest and AudioInputRequest on the embed.v_2.tasks.create method. The SDK accepts both the old and the new classes, so your existing code continues to work after you upgrade. Update it to clear the type-checker errors.

Before:

1
from twelvelabs.types import VideoInputRequest, MediaSource
2
3
task = client.embed.v_2.tasks.create(
4
    input_type="video",
5
    model_name="marengo3.0",
6
    video=VideoInputRequest(media_source=MediaSource(asset_id="<YOUR_ASSET_ID>")),
7
)

After:

1
from twelvelabs.types import AsyncVideoInputRequest, MediaSource
2
3
task = client.embed.v_2.tasks.create(
4
    input_type="video",
5
    model_name="marengo3.0",
6
    video=AsyncVideoInputRequest(media_source=MediaSource(asset_id="<YOUR_ASSET_ID>")),
7
)

Update your synchronous calls

Marengo 3.5 accepts a single input type on the synchronous method: multi_input. Replace the per-type input objects with a single combined input. Provide your text, your media sources, or both.

Marengo 3.0 accepts images as media sources. Marengo 3.5 also accepts video and audio.

1
from twelvelabs.types import TextInputRequest
2
3
response = client.embed.v_2.create(
4
    input_type="text",
5
    model_name="marengo3.0",
6
    text=TextInputRequest(input_text="<YOUR_TEXT>"),
7
)

1
from twelvelabs.types import MultiInputRequest
2
3
response = client.embed.v_2.create(
4
    input_type="multi_input",
5
    model_name="marengo3.5",
6
    multi_input=MultiInputRequest(input_text="<YOUR_TEXT>"),
7
)

Update your asynchronous calls

Set the model to Marengo 3.5. This version also accepts documents and images as input types.

Marengo 3.5 changes two defaults. For video, the embedding_option field defaults to ["visual", "audio"] instead of ["visual", "audio", "transcription"]. For audio, it defaults to ["audio"] instead of ["audio", "transcription"]. The transcription value requires Marengo 3.0. With Marengo 3.5, the audio value includes speech, music, and non-dialog audio.

Marengo 3.5 also changes the structure of the segmentation field. Place your settings in a temporal object. For details, see the Video embeddings guide.

1
task = client.embed.v_2.tasks.create(
2
    input_type="video",
3
    model_name="marengo3.0",
4
    video=AsyncVideoInputRequest(media_source=MediaSource(asset_id="<YOUR_ASSET_ID>")),
5
)

1
task = client.embed.v_2.tasks.create(
2
    input_type="video",
3
    model_name="marengo3.5",
4
    video=AsyncVideoInputRequest(media_source=MediaSource(asset_id="<YOUR_ASSET_ID>")),
5
)

Regenerate your embeddings

Marengo 3.5 embeddings are not compatible with Marengo 3.0 embeddings. To move to Marengo 3.5, regenerate your embeddings with Marengo 3.5.

Re-embed the same source content with Marengo 3.5. Keep the two sets of embeddings apart until the new set is complete. Then point your application at the new set. For step-by-step instructions, see the Embed content at scale guides.

Use new Marengo 3.5 features

The following examples show each new capability.

Composed queries with video and audio

Combine text with up to 10 media sources into a single embedding. Marengo 3.0 accepts images as media sources. Marengo 3.5 also accepts video and audio. Reference a media source from your text with the <@name> format.

1
from twelvelabs.types import MultiInputRequest, MultiInputMediaSource
2
3
response = client.embed.v_2.create(
4
    input_type="multi_input",
5
    model_name="marengo3.5",
6
    multi_input=MultiInputRequest(
7
        input_text="A crowd reacting like <@clip>",
8
        media_sources=[
9
            MultiInputMediaSource(
10
                name="clip",
11
                media_type="video",
12
                asset_id="<YOUR_ASSET_ID>",
13
            ),
14
        ],
15
    ),
16
)

Document embeddings

Create embeddings from a PDF file. Set the embedding_scope field to the local scope for one embedding per page. Documents require the asynchronous method.

1
from twelvelabs.types import AsyncDocumentInputRequest, MediaSource
2
3
task = client.embed.v_2.tasks.create(
4
    input_type="document",
5
    model_name="marengo3.5",
6
    document=AsyncDocumentInputRequest(
7
        media_source=MediaSource(asset_id="<YOUR_ASSET_ID>"),
8
        embedding_scope=["local"],
9
    ),
10
)

Time-based metadata fusion

Fold your own time-aligned text, such as a stats feed, into the fused embedding of the segments it overlaps in time. This capability requires the fused_embedding value in the embedding_type field, and it applies to audio and video on the asynchronous method.

1
from twelvelabs.types import AsyncVideoInputRequest, MediaSource, TimeBasedMetadataEntry
2
3
task = client.embed.v_2.tasks.create(
4
    input_type="video",
5
    model_name="marengo3.5",
6
    video=AsyncVideoInputRequest(
7
        media_source=MediaSource(asset_id="<YOUR_ASSET_ID>"),
8
        embedding_option=["visual", "audio"],
9
        embedding_type=["separate_embedding", "fused_embedding"],
10
        time_based_metadata=[
11
            TimeBasedMetadataEntry(start=42.3, end=45.0, text="<YOUR_TEXT>"),
12
        ],
13
    ),
14
)

Embedding uncertainty

Set the embedding_uncertainty parameter to true to receive a per-dimension uncertainty vector alongside each embedding. A higher value shows lower confidence in that dimension.

On the synchronous method, set this parameter to true only when your request embeds text only, or media only. On the asynchronous method with audio or video input, exclude the asset scope from the embedding_scope field. For details, see the Embed a query and Embed content at scale guides.

Token usage

Responses from Marengo 3.5 include a new object named usage that shows the token counts for your request.

Automatic truncation

Set the auto_truncate parameter to true to truncate text above the 2,000-token limit instead of receiving a 400 error. When the platform truncates your text, it sets the usage.truncated field to true in the response.