Create async embeddings

The Embed.V2.Tasks interface provides methods to create embeddings asynchronously for audio and video content. Use this class for processing files longer than 10 minutes.

Note

This interface only supports Marengo version 3.0 or newer.

When to use this interface:

  • Process audio or video files longer than 10 minutes
  • Process files up to 4 hours in duration

Video:

  • Minimum duration: 4 seconds
  • Maximum duration: 4 hours
  • Maximum file size: 4 GB
  • Formats: FFmpeg supported formats
  • Resolution: 360x360 to 3840x2160 pixels
  • Aspect ratio: Between 1:1 and 1:2.4, or between 2.4:1 and 1:1

Audio:

  • Minimum duration: 4 seconds
  • Maximum duration: 4 hours
  • Maximum file size: 2 GB
  • Formats: WAV (uncompressed), MP3 (lossy), FLAC (lossless)

Creating embeddings asynchronously requires three steps:

  1. Create a task using the create method. The platform returns a task ID.
  2. Poll for the status of the task using the retrieve method. Wait until the status is ready.
  3. Retrieve the embeddings from the response when the status is ready using the retrieve method.

Methods

List embedding tasks

Description: This method returns a list of the async embedding tasks in your account. The platform returns your async embedding tasks sorted by creation date, with the newest at the top of the list.

Function signature and example:

1list(
2 request?: TwelvelabsApi.embed.v2.TasksListRequest,
3 requestOptions?: Tasks.RequestOptions
4): Promise<core.Page<TwelvelabsApi.MediaEmbeddingTask>>

Parameters:

NameTypeRequiredDescription
startedAtstringNoRetrieve the embedding tasks that were created after the given date and time, expressed in the RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”).
endedAtstringNoRetrieve the embedding tasks that were created before the given date and time, expressed in the RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”).
statusstringNoFilter the embedding tasks by their current status. Values: processing, ready, or failed.
pagenumberNoA number that identifies the page to retrieve. Default: 1.
pageLimitnumberNoThe number of items to return on each page. Default: 10. Max: 50.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns a Promise that resolves to a Page<MediaEmbeddingTask> object that allows you to iterate through the paginated task results.

The Page<T> class contains the following properties and methods:

NameTypeDescription
dataT[]An array containing the current page of items.
hasNextPage()booleanReturns whether there is a next page to load.
getNextPage()Promise<Page<T>>Retrieves the next page and returns the updated Page object.
[Symbol.asyncIterator]()AsyncIterator<T>Allows iteration through all items across all pages using for await loops.

The MediaEmbeddingTask interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the embedding task.
modelNamestringThe name of the video understanding model the platform used to create the embedding.
statusstringA string indicating the status of the embedding task. Values: processing, ready, or failed.
createdAtDateThe date and time when the task was created.
updatedAtDateThe date and time when the task was last updated.
videoEmbeddingTwelvelabsApi.MediaEmbeddingTaskVideoEmbeddingAn object containing the metadata associated with the video embedding.
audioEmbeddingTwelvelabsApi.MediaEmbeddingTaskAudioEmbeddingAn object containing the metadata associated with the audio embedding.

API Reference: List async embedding tasks

Create an async embedding task

Description: This method creates embeddings for audio and video content asynchronously.

Function signature and example:

1create(
2 request: TwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequest,
3 requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.embed.v2.TasksCreateResponse>

Parameters:

NameTypeRequiredDescription
inputTypeTwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequestInputTypeYesThe type of content for the embeddings. Values: audio, video.
modelNamestringYesThe model you wish to use. Example: marengo3.0.
audioTwelvelabsApi.AudioInputRequestNoAudio input configuration. Required when inputType is audio. See AudioInputRequest for details.
videoTwelvelabsApi.VideoInputRequestNoVideo input configuration. Required when inputType is video. See VideoInputRequest for details.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

AudioInputRequest

The AudioInputRequest interface specifies configuration for processing audio content. Required when inputType is audio.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the audio file. See MediaSource for details.
startSecnumberNoThe start time in seconds for processing the audio file. Use this parameter to process a portion of the audio file starting from a specific time. Default: 0 (start from the beginning).
endSecnumberNoThe end time in seconds for processing the audio file. Use this parameter to process a portion of the audio file ending at a specific time. The end time must be greater than the start time. Default: End of the audio file.
segmentationTwelvelabsApi.AudioSegmentationNoSpecifies how the platform divides the audio into segments. See AudioSegmentation for details.
embeddingOptionTwelvelabsApi.AudioInputRequestEmbeddingOptionItem[]NoThe types of embeddings you wish to generate. Values:
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech

You can specify multiple options to generate different types of embeddings for the same audio.
embeddingScopeTwelvelabsApi.AudioInputRequestEmbeddingScopeItem[]NoThe scope for which you wish to generate embeddings. Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire audio file

You can specify multiple scopes to generate embeddings at different levels.

VideoInputRequest

The VideoInputRequest interface specifies configuration for processing video content. Required when inputType is video.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the video file. See MediaSource for details.
startSecnumberNoThe start time in seconds for processing the video file. Use this parameter to process a portion of the video file starting from a specific time. Default: 0 (start from the beginning).
endSecnumberNoThe end time in seconds for processing the video file. Use this parameter to process a portion of the video file ending at a specific time. The end time must be greater than the start time. Default: End of the video file.
segmentationTwelvelabsApi.VideoSegmentationNoSpecifies how the platform divides the video into segments. See VideoSegmentation for details.
embeddingOptionTwelvelabsApi.VideoInputRequestEmbeddingOptionItem[]NoThe types of embeddings to generate for the video. Values:
- visual: Generates embeddings based on visual content (scenes, objects, actions)
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech

You can specify multiple options to generate different types of embeddings for the same video. Default: ["visual", "audio", "transcription"].
embeddingScopeTwelvelabsApi.VideoInputRequestEmbeddingScopeItem[]NoThe scope for which you wish to generate embeddings. Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire video file

You can specify multiple scopes to generate embeddings at different levels. Default: ["clip", "asset"].

MediaSource

The MediaSource interface specifies the source of the media file. Provide exactly one of the following:

NameTypeRequiredDescription
base64StringstringNoThe base64-encoded media data.
urlstringNoThe publicly accessible URL of the media file. Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported.
assetIdstringNoThe unique identifier of an asset from a direct or multipart upload.

AudioSegmentation

The AudioSegmentation interface specifies how the platform divides the audio into segments using fixed-length intervals.

NameTypeRequiredDescription
strategy"fixed"YesThe segmentation strategy. Value: fixed.
fixedTwelvelabsApi.AudioSegmentationFixedYesConfiguration for fixed segmentation. See AudioSegmentationFixed for details.

AudioSegmentationFixed

The AudioSegmentationFixed interface configures fixed-length segmentation for audio.

NameTypeRequiredDescription
durationSecnumberYesThe duration in seconds for each segment. The platform divides the audio into segments of this exact length. The final segment may be shorter if the audio duration is not evenly divisible.

Example: With durationSec: 5, a 12-second audio file produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentation

The VideoSegmentation type specifies how the platform divides the video into segments. Use one of the following:

Fixed segmentation: Divides the video into equal-length segments:

NameTypeRequiredDescription
strategy"fixed"YesThe segmentation strategy. Value: fixed.
fixedTwelvelabsApi.VideoSegmentationFixedFixedYesConfiguration for fixed segmentation. See VideoSegmentationFixedFixed for details.

Dynamic segmentation: Divides the video into adaptive segments based on scene changes:

NameTypeRequiredDescription
strategy"dynamic"YesThe segmentation strategy. Value: dynamic.
dynamicTwelvelabsApi.VideoSegmentationDynamicDynamicYesConfiguration for dynamic segmentation. See VideoSegmentationDynamicDynamic for details.

VideoSegmentationFixedFixed

The VideoSegmentationFixedFixed interface configures fixed-length segmentation for video.

NameTypeRequiredDescription
durationSecnumberYesThe duration in seconds for each segment. The platform divides the video into segments of this exact length. The final segment may be shorter if the video duration is not evenly divisible.

Example: With durationSec: 5, a 12-second video produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentationDynamicDynamic

The VideoSegmentationDynamicDynamic interface configures dynamic segmentation for video based on scene changes.

NameTypeRequiredDescription
minDurationSecnumberYesThe minimum duration in seconds for each segment. The platform divides the video into segments that are at least this long. Segments adapt to scene changes and content boundaries and may be longer than the minimum.

Example: With minDurationSec: 3, segments might be: [0-3.2s], [3.2-7.8s], [7.8-12.1s].

Return value: Returns a Promise that resolves to a TasksCreateResponse object containing the task details.

The TasksCreateResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the embedding task.
status"processing"The initial status of the embedding task. Value: processing.
dataTwelvelabsApi.EmbeddingData[]Array of embedding results (only when status is ready).

API Reference: Create an async embedding task

Retrieve task status and results

Description: This method retrieves the status and the results of an async embedding task.

Task statuses:

  • processing: The platform is creating the embeddings.
  • ready: Processing is complete. Embeddings are available in the response.
  • failed: The task failed. Embeddings were not created.

Invoke this method repeatedly until the status field is ready. When status is ready, use the embeddings from the response.

Function signature and example:

1retrieve(
2 taskId: string,
3 requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.EmbeddingTaskResponse>

Parameters:

NameTypeRequiredDescription
taskIdstringYesThe unique identifier of the embedding task.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns a Promise that resolves to an EmbeddingTaskResponse object containing the task status and results.

The EmbeddingTaskResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the embedding task.
statusTwelvelabsApi.EmbeddingTaskResponseStatusThe current status of the task. Values:
- processing: The platform is creating the embeddings
- ready: Processing is complete. Embeddings are available in the data field
- failed: The task failed. The data field is null
createdAtDateThe date and time when the task was created.
updatedAtDateThe date and time when the task was last updated.
dataTwelvelabsApi.EmbeddingData[]An array of embedding results. The platform returns this field when status is ready.
metadataTwelvelabsApi.EmbeddingTaskMediaMetadataMetadata about the embedding task.

For details about the EmbeddingData class, see the Create an async embedding task section.

API Reference: Retrieve embedding task

API Reference: Retrieve task status and results