Node.js SDKModels and searchCreate embeddings v2

Create sync embeddings

The Embed.V2 interface provides methods to create embeddings synchronously for multimodal content. It returns embeddings immediately in the response.

Methods

Create sync embeddings

Description: This method synchronously creates embeddings for multimodal content and returns the results immediately in the response.

Use this method to embed a query for retrieving matching content. With Marengo 3.5, audio and video can be up to 30 seconds. With Marengo 3.0, they can be up to 10 minutes. For longer content, use the embed.v2.tasks.create method instead.

The content this method accepts depends on the model. With Marengo 3.5, this method accepts only the multiInput input type; provide text, images, audio, or video as media sources. With Marengo 3.0, use the individual input types. For the formats, resolutions, file sizes, and duration limits each model accepts, see the input requirements for Marengo 3.5 or Marengo 3.0.

Note

This method is rate-limited. With Marengo 3.5, the platform counts input tokens for each type of content. A request can exceed a limit before you see an error. For details, see Input token limits for embedding.

Function signature and example:

1create(
2    request: TwelvelabsApi.embed.CreateEmbeddingsRequest,
3    requestOptions?: V2.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.EmbeddingSuccessResponse>

Parameters

NameTypeRequiredDescription
requestTwelvelabsApi.embed.CreateEmbeddingsRequestYesParameters for creating embeddings.
requestOptionsV2.RequestOptionsNoRequest-specific configuration.

The TwelvelabsApi.embed.CreateEmbeddingsRequest interface contains the following properties:

NameTypeRequiredDescription
inputTypeTwelvelabsApi.embed.CreateEmbeddingsRequestInputTypeYesThe type of content for the embeddings. Values:
- multi_input: Text and up to 10 media sources, combined into a single embedding. To reference a specific media source from your text, use a placeholder in the format &lt;@name&gt;, where name matches the name field of a media source. Marengo 3.5 accepts images, video, and audio as media sources. Marengo 3.0 accepts images.
- audio: An audio file. Requires Marengo 3.0.
- video: A video file. Requires Marengo 3.0.
- image: An image file. Requires Marengo 3.0.
- text: Text input. Requires Marengo 3.0.
- text_image: Text and an image. Requires Marengo 3.0.
modelNameTwelvelabsApi.embed.CreateEmbeddingsRequestModelNameYesThe embedding model to use. Values:
- marengo3.5: For details about this version, see the Marengo 3.5 page.
- marengo3.0: For details about this version, see the Marengo 3.0 page.
textTwelvelabsApi.TextInputRequestNoText input configuration. Required when inputType is text. See TextInputRequest for details.
imageTwelvelabsApi.ImageInputRequestNoImage input configuration. Required when inputType is image. See ImageInputRequest for details.
textImageTwelvelabsApi.TextImageInputRequestNoCombined text and image input configuration. Required when inputType is text_image. See TextImageInputRequest for details.
audioTwelvelabsApi.AudioInputRequestNoAudio input configuration. Required when inputType is audio. See AudioInputRequest for details.
videoTwelvelabsApi.VideoInputRequestNoVideo input configuration. Required when inputType is video. See VideoInputRequest for details.
multiInputTwelvelabsApi.MultiInputRequestNoText and media source configuration. Required when inputType is multi_input. See MultiInputRequest for details.
autoTruncatebooleanNoControls the behavior of the platform when the text in your request exceeds 2,000 tokens. Requires Marengo 3.5.
Values:
- false: Return a 400 error.
- true: Truncate your text to fit the limit, and set the usage.truncated field to true in the response.
embeddingUncertaintybooleanNoSet this parameter to true to receive a data[].embeddingUncertainty field in the response, representing a per-dimension uncertainty vector with the same length as the embedding array. A higher value shows lower confidence in that dimension. Requires Marengo 3.5.
Set this parameter to true only when your request embeds text only, or media only. Requests that combine text with media sources return a 400 error.

TextInputRequest

The TwelvelabsApi.TextInputRequest interface specifies the configuration for processing text content. Required when inputType is text.

NameTypeRequiredDescription
inputTextstringYesThe text for which you wish to create an embedding. The maximum length is 500 tokens.

ImageInputRequest

The TwelvelabsApi.ImageInputRequest interface specifies the configuration for processing image content. Required when inputType is image.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the image file. See MediaSource for details.

TextImageInputRequest

The TwelvelabsApi.TextImageInputRequest interface specifies the configuration for processing combined text and image content. Required when inputType is text_image.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the image file. See MediaSource for details.
inputTextstringYesThe text for which you wish to create an embedding. The maximum length is 500 tokens.

AudioInputRequest

The TwelvelabsApi.AudioInputRequest interface specifies the configuration for processing audio content. Required when inputType is audio.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the audio file. See MediaSource for details.
startSecnumberNoThe start time in seconds for processing the audio file.
Use this parameter to process a portion of the audio file starting from a specific time.
Default: 0 (start from the beginning).
endSecnumberNoThe end time in seconds for processing the audio file.
Use this parameter to process a portion of the audio file ending at a specific time. The end time must be greater than the start time.
Default: End of the audio file
segmentationTwelvelabsApi.AudioSegmentationNoSpecifies how the platform divides the audio into segments. See AudioSegmentation for details.
embeddingOptionTwelvelabsApi.AudioInputRequestEmbeddingOptionItem[]NoThe types of embeddings you wish to generate.
Values:
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech
You can specify multiple values to generate different types of embeddings for the same audio.
Default: ["audio", "transcription"]
embeddingScopeTwelvelabsApi.AudioInputRequestEmbeddingScopeItem[]NoThe scope for which you wish to generate embeddings.
Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire audio file
You can specify multiple scopes to generate embeddings at different levels.
Default: ["clip", "asset"]
embeddingTypeTwelvelabsApi.AudioInputRequestEmbeddingTypeItem[]NoSpecifies how to structure the embedding. Include this parameter only when the embeddingOption parameter contains at least two values.
Values:
- separate_embedding: Returns separate embeddings for each modality specified in the embeddingOption parameter.
- fused_embedding: Returns a single combined embedding that integrates all modalities into one vector.
Specify both values to receive separate and fused embeddings in the same response.
Default: separate_embedding.

VideoInputRequest

The TwelvelabsApi.VideoInputRequest interface specifies the configuration for processing video content. Required when inputType is video.

NameTypeRequiredDescription
mediaSourceTwelvelabsApi.MediaSourceYesSpecifies the source of the video file. See MediaSource for details.
startSecnumberNoThe start time in seconds for processing the video file.
Use this parameter to process a portion of the video file starting from a specific time.
Default: 0 (start from the beginning)
endSecnumberNoThe end time in seconds for processing the video file.
Use this parameter to process a portion of the video file ending at a specific time. The end time must be greater than the start time.
Default: End of the video file
segmentationTwelvelabsApi.VideoSegmentationNoSpecifies how the platform divides the video into segments. See VideoSegmentation for details.
embeddingOptionTwelvelabsApi.VideoInputRequestEmbeddingOptionItem[]NoThe types of embeddings to generate for the video.
Values:
- visual: Generates embeddings based on visual content (scenes, objects, actions)
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech
You can specify multiple values to generate different types of embeddings for the same video.
Default: ["visual", "audio", "transcription"]
embeddingScopeTwelvelabsApi.VideoInputRequestEmbeddingScopeItem[]NoThe scope for which you wish to generate embeddings.
Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire video file. Use this scope for videos up to 10-30 seconds to maintain optimal performance.
You can specify multiple scopes to generate embeddings at different levels.
Default: ["clip", "asset"]
embeddingTypeTwelvelabsApi.VideoInputRequestEmbeddingTypeItem[]NoSpecifies how to structure the embedding. Include this parameter only when embeddingOption contains at least two values.
Values:
- separate_embedding: Returns separate embeddings per modality specified in the embeddingOption field
- fused_embedding: Returns a single embedding that combines all modalities into one vector
Specify both values to receive separate and fused embeddings in the same response.
Default: separate_embedding.

MultiInputRequest

The MultiInputRequest class specifies the configuration for processing text and media sources. Required when inputType is multi_input.

Marengo 3.5 accepts images, video, and audio as media sources. Marengo 3.0 accepts images.

Include text in the input_text field when you combine media sources of different types. For example, a request that combines an image and a video returns a 400 error without text. Media sources of the same type do not require text.

Marengo 3.5 accepts images, video, and audio as media sources. Marengo 3.0 accepts images.

NameTypeRequiredDescription
inputTextstringNoText to include in the embedding. With Marengo 3.5, the text cannot exceed 2,000 tokens. Use the autoTruncate parameter to control the behavior of the platform when your text exceeds it.
Usage options:
- Provide text without media sources to create a text-only embedding.
- Combine text with media sources to add context. Example: “A person cooking.”
- Use media source references to describe relationships between specific media sources. The format is &lt;@name&gt;, where name matches the name field of a media source. Example: “A person wearing <@outfit> and holding <@accessory>.”
- Omit this field to create an embedding from media sources only.
mediaSourcesTwelvelabsApi.MultiInputMediaSource[]NoAn array of up to 10 media sources to include in the embedding. Omit it to create a text-only embedding from the inputText field. The platform processes media sources in the order they appear in the array. If you use media source references in the inputText parameter, each must have a corresponding media source with a matching name field. If a reference has no match, the request fails.

MediaSource

The TwelvelabsApi.MediaSource interface specifies the source of the media file. Provide exactly one of the following:

NameTypeRequiredDescription
base64StringstringNoThe base64-encoded media data.
urlstringNoThe publicly accessible URL of the media file. Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported.
assetIdstringNoThe unique identifier of an asset from a direct or multipart upload. The asset status must be ready. Use assets.retrieve to check the status.

MultiInputMediaSource

A class specifying a media source for multi-input embeddings. You must provide exactly one of the url, base64String, or assetId fields. With Marengo 3.5, each media source can be up to 32 MB, whichever of the three fields you use. Audio and video can be up to 30 seconds. Content above either limit returns a 400 error.

NameTypeRequiredDescription
namestringNoThe unique identifier for this media source.
This field is required when inputText references this media source.
mediaTypeTwelvelabsApi.MultiInputMediaSourceMediaTypeNoThe type of media.
Values:
- image: An image file. Works with both Marengo 3.0 and Marengo 3.5.
- video: A video file. Requires Marengo 3.5.
- audio: An audio file. Requires Marengo 3.5.
urlstringNoThe publicly accessible URL of the media file. Use direct links to raw files. Media hosting platforms and cloud storage sharing links are not supported.
base64StringstringNoThe base64-encoded media data.
assetIdstringNoThe unique identifier of an asset from a direct or multipart upload.

AudioSegmentation

The TwelvelabsApi.AudioSegmentation interface specifies how the platform divides the audio into segments using fixed-length intervals.

NameTypeRequiredDescription
strategy"fixed"YesThe segmentation strategy. Value: fixed.
fixedTwelvelabsApi.AudioSegmentationFixedYesConfiguration for fixed segmentation.
This object is required when the strategy field is fixed. See AudioSegmentationFixed for details.

AudioSegmentationFixed

The TwelvelabsApi.AudioSegmentationFixed interface configures fixed-length segmentation for audio.

NameTypeRequiredDescription
durationSecnumberYesThe duration in seconds for each segment. The platform divides the audio into segments of this exact length. The final segment may be shorter if the audio duration is not evenly divisible.
Min: 2.
Max: 10.
Example: With duration_sec: 5, a 12-second audio file produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentation

The TwelvelabsApi.VideoSegmentation type specifies how the platform divides the video into segments. Use one of the following:

Fixed segmentation: Divides the video into equal-length segments:

NameTypeRequiredDescription
strategy"fixed"YesThe segmentation strategy. Value: fixed.
fixedTwelvelabsApi.VideoSegmentationFixedFixedYesConfiguration for fixed segmentation. See VideoSegmentationFixedFixed for details.

Dynamic segmentation: Divides the video into adaptive segments based on scene changes:

NameTypeRequiredDescription
strategy"dynamic"YesThe segmentation strategy. Value: dynamic.
dynamicTwelvelabsApi.VideoSegmentationDynamicDynamicYesConfiguration for dynamic segmentation. See VideoSegmentationDynamicDynamic for details.

VideoSegmentationFixedFixed

The TwelvelabsApi.VideoSegmentationFixedFixed interface configures fixed-length segmentation for video.

NameTypeRequiredDescription
durationSecnumberYesThe duration in seconds for each segment.
The platform divides the video into segments of this exact length. The final segment may be shorter if the video duration is not evenly divisible.
Min: 2.
Max: 10.
Example: With duration_sec: 5, a 12-second video produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentationDynamicDynamic

The TwelvelabsApi.VideoSegmentationDynamicDynamic interface configures dynamic segmentation for video based on scene changes.

NameTypeRequiredDescription
minDurationSecnumberYesThe minimum duration in seconds for each segment.
The platform divides the video into segments that are at least this long. Segments adapt to scene changes and content boundaries and may be longer than the minimum.
Min: 2.
Max: 5.
Example: With min_duration_sec: 3, segments might be: [0-3.2s], [3.2-7.8s], [7.8-12.1s]

Return value

Returns an HttpResponsePromise that resolves to a TwelvelabsApi.EmbeddingSuccessResponse object containing the embedding results.

The TwelvelabsApi.EmbeddingSuccessResponse interface contains the following properties:

NameTypeDescription
dataTwelvelabsApi.EmbeddingData[]Array of embedding results.
usageTwelvelabsApi.EmbeddingUsageToken counts for the request. Only Marengo 3.5 returns this field. See EmbeddingUsage for details.
metadataTwelvelabsApi.EmbeddingMediaMetadataMetadata for the media input. Available for the image, text_image, audio, video, and multi_input input types. See EmbeddingMediaMetadata for details.

The TwelvelabsApi.EmbeddingData interface contains the following properties:

NameTypeDescription
embeddingnumber[]The embedding vector for the content.
embeddingUncertaintyOptional<number[]>A per-dimension uncertainty vector with the same length as the embedding array. A higher value shows lower confidence in that dimension. Present when the request sets embeddingUncertainty: true. Only Marengo 3.5 returns this field.
embeddingOptionOptional<TwelvelabsApi.EmbeddingDataEmbeddingOption>The modality used to generate this embedding.
Values:
- visual: Embedding based on visual content (a video, a page of a PDF file, or an image embedded asynchronously).
- audio: Embedding based on audio content.
- transcription: Embedding based on transcribed speech. Returned only for content embedded with Marengo 3.0.
- text: The platform does not return this value.
- fused: Embedding based on a combination of the modalities specified in the request. The platform returns this embedding only for video and audio input, and only when the embeddingType parameter includes the fused_embedding value.
- null: For text embeddings and images embedded synchronously.
embeddingScopeOptional<TwelvelabsApi.EmbeddingDataEmbeddingScope>The scope for which the embedding was generated.
Values:
- clip: Embedding for a segment. For video and audio input, one embedding per detected segment.
- page: Embedding for one page of a document. The platform returns this value only for PDF files embedded asynchronously.
- asset: Embedding for the entire file. For video and audio input, use this scope for content up to 10-30 seconds to maintain optimal performance.
- null: For text embeddings and images embedded synchronously.
startSecOptional<number>The start time in seconds for this segment. This field is null for text and image embeddings.
endSecOptional<number>The end time in seconds for this segment. This field is null for text and image embeddings.
startPageNumberOptional<number>The first page this embedding covers, counting from 1. The platform returns this field only for page-level embeddings of a PDF file, and null in every other case.
endPageNumberOptional<number>The last page this embedding covers, counting from 1 and including that page. This field matches the startPageNumber field when the embedding covers a single page. The platform returns this field only for page-level embeddings of a PDF file, and null in every other case.

EmbeddingUsage

The TwelvelabsApi.EmbeddingUsage interface contains token counts for the request. Only Marengo 3.5 returns this object.

NameTypeDescription
inputTokensRecord<string, number>The number of tokens the request used. Each key names a type of content the request processed, and each value is the token count for that content.
truncatedbooleanWhether the input was truncated to fit within the token limit.

EmbeddingMediaMetadata

The TwelvelabsApi.EmbeddingMediaMetadata type provides metadata for the media input. Available for the image, text_image, audio, video, and multi_input input types. The inputType field selects one variant:

Image: Metadata for image embeddings.

NameTypeDescription
inputType"image"The type of the input content. Value: image.
inputUrlOptional<string>The publicly accessible URL for the image file.
inputFilenameOptional<string>The name of the image file.

Text and image: Metadata for text-image embeddings.

NameTypeDescription
inputType"text_image"The type of the input content. Value: text_image.
inputUrlOptional<string>The publicly accessible URL for the image file.
inputFilenameOptional<string>The name of the image file.

Audio: Metadata for audio embeddings.

NameTypeDescription
inputType"audio"The type of the input content. Value: audio.
inputUrlOptional<string>The publicly accessible URL for the audio file.
inputFilenameOptional<string>The name of the audio file.
embeddingOptionsstring[]The embeddingOption values used to generate the embedding.
embeddingScopesTwelvelabsApi.EmbeddingAudioMetadataEmbeddingScopesItem[]The embeddingScope values used to generate the embedding.
durationnumberThe duration of the audio in seconds.
startOffsetSecOptional<number>The start offset in seconds.
endOffsetSecOptional<number>The end offset in seconds.

Video: Metadata for video embeddings.

NameTypeDescription
inputType"video"The type of the input content. Value: video.
inputUrlOptional<string>The publicly accessible URL for the video file.
inputFilenameOptional<string>The name of the video file.
clipLengthOptional<number>Length of each video clip in seconds. Only available for fixed segmentation.
embeddingScopesTwelvelabsApi.EmbeddingVideoMetadataEmbeddingScopesItem[]The embeddingScope values used to generate the embedding.
embeddingOptionsstring[]The embeddingOption values used to generate the embedding.
durationnumberThe duration of the video in seconds.
startOffsetSecOptional<number>The start offset in seconds.
endOffsetSecOptional<number>The end offset in seconds.

Multi-input: Metadata for multi-input embeddings.

NameTypeDescription
inputType"multi_input"The type of the input content. Value: multi_input.

API Reference

Create sync embeddings