For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sample appsIntegrationsDiscordPlaygroundDevEx repo
GuidesSDK ReferenceAPI Reference
GuidesSDK ReferenceAPI Reference
  • Python SDK
    • The TwelveLabs class
    • Manage indexes
    • Upload content
    • Index content
    • Manage videos
    • Manage entities
    • Search
    • Create embeddings v2
    • Create embeddings v1
    • Analyze and segment videos
  • Node.js SDK
    • The TwelveLabs class
    • Manage indexes
    • Upload content
    • Index content
    • Manage videos
    • Manage entities
    • Search
    • Create embeddings v2
      • Create sync embeddings
      • Create async embeddings
    • Create embeddings v1
    • Analyze and segment videos
LogoLogo
Sample appsIntegrationsDiscordPlaygroundDevEx repo
On this page
  • Methods
  • List embedding tasks
  • Parameters
  • Return value
  • API Reference
  • Create an async embedding task
  • Parameters
  • AudioInputRequest
  • VideoInputRequest
  • MediaSource
  • AudioSegmentation
  • AudioSegmentationFixed
  • VideoSegmentation
  • VideoSegmentationFixedFixed
  • VideoSegmentationDynamicDynamic
  • Return value
  • API Reference
  • Retrieve task status and results
  • Parameters
  • Return value
  • API Reference
Node.js SDKCreate embeddings v2

Create async embeddings

Was this page helpful?
Previous

Create embeddings v1

Next
Built with

The Embed.V2.Tasks interface provides methods to create embeddings asynchronously for audio and video content. Use this class for processing files longer than 10 minutes.

When to use this interface:

  • Process audio or video files longer than 10 minutes
  • Process files up to 4 hours in duration
Input requirements

Video:

  • Minimum duration: 4 seconds
  • Maximum duration: 4 hours
  • Maximum file size: 4 GB
  • Formats: FFmpeg supported formats
  • Resolution: 360x360 to 5184x2160 pixels
  • Aspect ratio: Between 1:1 and 1:2.4, or between 2.4:1 and 1:1

Audio:

  • Minimum duration: 4 seconds
  • Maximum duration: 4 hours
  • Maximum file size: 4 GB
  • Formats: WAV (uncompressed), MP3 (lossy), FLAC (lossless)

Creating embeddings asynchronously requires three steps:

  1. Create a task using the create method. The platform returns a task ID.
  2. Poll for the status of the task using the retrieve method. Wait until the status is ready.
  3. Retrieve the embeddings from the response when the status is ready using the retrieve method.

Methods

List embedding tasks

Description: This method returns a list of the async embedding tasks in your account. The platform returns your async embedding tasks sorted by creation date, with the newest at the top of the list.

Function signature and example:

1list(
2    request?: TwelvelabsApi.embed.v2.TasksListRequest,
3    requestOptions?: Tasks.RequestOptions
4): Promise<core.Page<TwelvelabsApi.MediaEmbeddingTask>>

Parameters

NameTypeRequiredDescription
requestTwelvelabsApi.embed.v2.TasksListRequestYesParameters for listing embedding tasks.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

The TwelvelabsApi.embed.v2.TasksListRequest interface has the following properties:

NameTypeRequiredDescription
startedAtstringNoRetrieve the embedding tasks that were created after the given date and time, expressed in the RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”).
endedAtstringNoRetrieve the embedding tasks that were created before the given date and time, expressed in the RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”).
statusstringNoFilter the embedding tasks by their current status. Values: processing, ready, or failed.
pagenumberNoA number that identifies the page to retrieve. Default: 1.
pageLimitnumberNoThe number of items to return on each page. Default: 10. Max: 50.

Return value

Returns a Promise that resolves to a Page<TwelvelabsApi.MediaEmbeddingTask> object that allows you to iterate through the paginated task results.

The Page<T> class contains the following properties and methods:

NameTypeDescription
dataT[]An array containing the current page of items.
hasNextPage()booleanReturns whether there is a next page to load.
getNextPage()Promise<Page<T>>Retrieves the next page and returns the updated Page object.
Symbol.asyncIteratorAsyncIterator<T>Allows iteration through all items across all pages using for await loops.

The TwelvelabsApi.MediaEmbeddingTask interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the embedding task.
modelNamestringThe name of the video understanding model the platform used to create the embedding.
statusstringA string indicating the status of the embedding task. Values: processing, ready, or failed.
createdAtDateThe date and time when the task was created.
updatedAtDateThe date and time when the task was last updated.
videoEmbeddingTwelvelabsApi.MediaEmbeddingTaskVideoEmbeddingAn object containing the metadata associated with the video embedding.
audioEmbeddingTwelvelabsApi.MediaEmbeddingTaskAudioEmbeddingAn object containing the metadata associated with the audio embedding.

API Reference

List async embedding tasks

Create an async embedding task

Description: This method creates embeddings for audio and video content asynchronously.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1create(
2    request: TwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequest,
3    requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.embed.v2.TasksCreateResponse>

Parameters

NameTypeRequiredDescription
requestTwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequestYesParameters for creating an async embedding task.
requestOptionsIndexes.RequestOptionsNoRequest-specific configuration.

The TwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequest interface has the following properties:

NameTypeRequiredDescription
inputTypeTwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequestInputTypeYesThe type of content for the embeddings. Values: audio, video.
modelNameTwelvelabsApi.embed.v2.CreateAsyncEmbeddingRequestModelNameYesThe video understanding model you wish to use. Value: marengo3.0.
audioTwelvelabsApi.AudioInputRequestNoAudio input configuration. Required when inputType is audio. See AudioInputRequest for details.
videoTwelvelabsApi.VideoInputRequestNoVideo input configuration. Required when inputType is video. See VideoInputRequest for details.

AudioInputRequest

The TwelvelabsApi.AudioInputRequest interface specifies the configuration for processing audio content. Required when inputType is audio.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the audio file. See MediaSource for details.
startSecnumberNoThe start time in seconds for processing the audio file. Use this parameter to process a portion of the audio file starting from a specific time. Default: 0 (start from the beginning).
endSecnumberNoThe end time in seconds for processing the audio file. Use this parameter to process a portion of the audio file ending at a specific time. The end time must be greater than the start time. Default: End of the audio file.
segmentationTwelvelabsApi.AudioSegmentationNoSpecifies how the platform divides the audio into segments. See AudioSegmentation for details.
embeddingOptionTwelvelabsApi.AudioInputRequestEmbeddingOptionItem[]NoThe types of embeddings you wish to generate. Values:
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech

You can specify multiple values to generate different types of embeddings for the same audio.
embeddingScopeTwelvelabsApi.AudioInputRequestEmbeddingScopeItem[]NoThe scope for which you wish to generate embeddings. Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire audio file

You can specify multiple scopes to generate embeddings at different levels.
embeddingTypeTwelvelabsApi.AudioInputRequestEmbeddingTypeItem[]NoSpecifies how to structure the embedding. Use this parameter only when embeddingOption specifies two or more values. Values:
- separate_embedding: Returns separate embeddings per modality specified in embeddingOption.
- fused_embedding: Returns a single embedding that combines all modalities into one vector.
Specify both values to receive separate and fused embeddings in the same response.

Default: separate_embedding

VideoInputRequest

The TwelvelabsApi.VideoInputRequest interface specifies the configuration for processing video content. Required when inputType is video.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the video file. See MediaSource for details.
startSecnumberNoThe start time in seconds for processing the video file. Use this parameter to process a portion of the video file starting from a specific time. Default: 0 (start from the beginning).
endSecnumberNoThe end time in seconds for processing the video file. Use this parameter to process a portion of the video file ending at a specific time. The end time must be greater than the start time. Default: End of the video file.
segmentationTwelvelabsApi.VideoSegmentationNoSpecifies how the platform divides the video into segments. See VideoSegmentation for details.
embeddingOptionTwelvelabsApi.VideoInputRequestEmbeddingOptionItem[]NoThe types of embeddings to generate for the video. Values:
- visual: Generates embeddings based on visual content (scenes, objects, actions)
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech

You can specify multiple values to generate different types of embeddings for the same video.

Default: ["visual", "audio", "transcription"].
embeddingScopeTwelvelabsApi.VideoInputRequestEmbeddingScopeItem[]NoThe scope for which you wish to generate embeddings. Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire video file. Use this scope for videos up to 10-30 seconds to maintain optimal performance.

You can specify multiple scopes to generate embeddings at different levels.

Default: ["clip", "asset"].
embeddingTypeTwelvelabsApi.VideoInputRequestEmbeddingTypeItem[]NoSpecifies how to structure the embedding. Include this parameter only when embeddingOption contains at least two values. Values:
- separate_embedding: Returns separate embeddings per modality specified in embeddingOption.
- fused_embedding: Returns a single embedding that combines all modalities into one vector.
Specify both values to receive separate and fused embeddings in the same response.

Default: separate_embedding

MediaSource

The TwelvelabsApi.MediaSource interface specifies the source of the media file. Provide exactly one of the following:

NameTypeRequiredDescription
base64StringstringNoThe base64-encoded media data.
urlstringNoThe publicly accessible URL of the media file. Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported.
assetIdstringNoThe unique identifier of an asset from a direct or multipart upload. The asset status must be ready. Use assets.retrieve to check the status.

AudioSegmentation

The TwelvelabsApi.AudioSegmentation interface specifies how the platform divides the audio into segments using fixed-length intervals.

NameTypeRequiredDescription
strategyTwelvelabsApi.AudioSegmentationStrategyYesThe segmentation strategy. Value: fixed.
fixedTwelvelabsApi.AudioSegmentationFixedYesConfiguration for fixed segmentation. See AudioSegmentationFixed for details.

AudioSegmentationFixed

The TwelvelabsApi.AudioSegmentationFixed interface configures fixed-length segmentation for audio.

NameTypeRequiredDescription
durationSecnumberYesThe duration in seconds for each segment. The platform divides the audio into segments of this exact length. The final segment may be shorter if the audio duration is not evenly divisible.

Example: With durationSec: 5, a 12-second audio file produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentation

The TwelvelabsApi.VideoSegmentation type specifies how the platform divides the video into segments. Use one of the following:

Fixed segmentation: Divides the video into equal-length segments:

NameTypeRequiredDescription
strategy"fixed"YesThe segmentation strategy. Value: fixed.
fixedTwelvelabsApi.VideoSegmentationFixedFixedYesConfiguration for fixed segmentation. See VideoSegmentationFixedFixed for details.

Dynamic segmentation: Divides the video into adaptive segments based on scene changes:

NameTypeRequiredDescription
strategy"dynamic"YesThe segmentation strategy. Value: dynamic.
dynamicTwelvelabsApi.VideoSegmentationDynamicDynamicYesConfiguration for dynamic segmentation. See VideoSegmentationDynamicDynamic for details.

VideoSegmentationFixedFixed

The TwelvelabsApi.VideoSegmentationFixedFixed interface configures fixed-length segmentation for video.

NameTypeRequiredDescription
durationSecnumberYesThe duration in seconds for each segment. The platform divides the video into segments of this exact length. The final segment may be shorter if the video duration is not evenly divisible.

Example: With durationSec: 5, a 12-second video produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentationDynamicDynamic

The TwelvelabsApi.VideoSegmentationDynamicDynamic interface configures dynamic segmentation for video based on scene changes.

NameTypeRequiredDescription
minDurationSecnumberYesThe minimum duration in seconds for each segment. The platform divides the video into segments that are at least this long. Segments adapt to scene changes and content boundaries and may be longer than the minimum.

Example: With minDurationSec: 3, segments might be: [0-3.2s], [3.2-7.8s], [7.8-12.1s].

Return value

Returns an HttpResponsePromise that resolves to a TwelvelabsApi.embed.v2.TasksCreateResponse object containing the task details.

The TwelvelabsApi.embed.v2.TasksCreateResponse.TasksCreateResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the embedding task.
statusTasksCreateResponseStatusThe initial status of the embedding task. Value: processing.
dataTwelvelabsApi.EmbeddingData[]Array of embedding results (only when status is ready).

API Reference

Create an async embedding task

Retrieve task status and results

Description: This method retrieves the status and the results of an async embedding task.

Task statuses:

  • processing: The platform is creating the embeddings.
  • ready: Processing is complete. Embeddings are available in the response.
  • failed: The task failed. Embeddings were not created.

Invoke this method repeatedly until the status field is ready. When status is ready, use the embeddings from the response.

Function signature and example:

1retrieve(
2    taskId: string,
3    requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.EmbeddingTaskResponse>

Parameters

NameTypeRequiredDescription
taskIdstringYesThe unique identifier of the embedding task.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value

Returns an HttpResponsePromise that resolves to a TwelvelabsApi.EmbeddingTaskResponse object containing the task status and results.

The TwelvelabsApi.EmbeddingTaskResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the embedding task.
statusTwelvelabsApi.EmbeddingTaskResponseStatusThe current status of the task. Values:
- processing: The platform is creating the embeddings
- ready: Processing is complete. Embeddings are available in the data field
- failed: The task failed. The data field is null
createdAtDateThe date and time when the task was created.
updatedAtDateThe date and time when the task was last updated.
dataTwelvelabsApi.EmbeddingData[]An array of embedding results. The platform returns this field when status is ready.
metadataTwelvelabsApi.EmbeddingTaskMediaMetadataMetadata about the embedding task.

The TwelvelabsApi.EmbeddingData interface contains the following properties:

NameTypeDescription
embeddingnumber[]The embedding vector for the content.
embeddingOptionTwelvelabsApi.EmbeddingDataEmbeddingOptionThe type of embedding. Values:
- visual: Embedding based on visual content (video only)
-audio: Embedding based on audio content
-transcription: Embedding based on transcribed speech
- fused: Embedding based on a combination of the modalities specified in the request. The platform returns this embedding only for video and audio content, and only when the embeddingType parameter in the request includes fused_embedding
- null: For text and image embeddings
embeddingScopeTwelvelabsApi.EmbeddingDataEmbeddingScopeThe scope of the embedding. Values: clip, asset.
startSecnumberThe start time in seconds for this embedding segment.
endSecnumberThe end time in seconds for this embedding segment.

API Reference

Retrieve task status and results