Create sync embeddings

The EmbedClient.V2Client class provides methods to create embeddings synchronously for multimodal content. This endpoint returns embeddings immediately in the response.

Note

This class only supports Marengo version 3.0 or newer.

When to use this class:

  • Create embeddings for text, images, audio, or video content
  • Retrieve immediate results without waiting for background processing
  • Process audio or video content up to 10 minutes in duration

Do not use this class for:

Methods

Create sync embeddings

Description: This method synchronously creates embeddings for multimodal content and returns the results immediately in the response.

Text:

  • Maximum length: 500 tokens

Images:

  • Formats: JPEG, PNG
  • Minimum size: 128x128 pixels
  • Maximum file size: 5 MB

Audio and video:

  • Maximum duration: 10 minutes
  • Maximum file size for base64 encoded strings: 36 MB
  • Audio formats: WAV (uncompressed), MP3 (lossy), FLAC (lossless)
  • Video formats: FFmpeg supported formats
  • Video resolution: 360x360 to 3840x2160 pixels
  • Aspect ratio: Between 1:1 and 1:2.4, or between 2.4:1 and 1:1

Function signature and example:

1def create(
2 self,
3 *,
4 input_type: CreateEmbeddingsRequestInputType,
5 model_name: str,
6 text: typing.Optional[TextInputRequest] = OMIT,
7 image: typing.Optional[ImageInputRequest] = OMIT,
8 text_image: typing.Optional[TextImageInputRequest] = OMIT,
9 audio: typing.Optional[AudioInputRequest] = OMIT,
10 video: typing.Optional[VideoInputRequest] = OMIT,
11 request_options: typing.Optional[RequestOptions] = None,
12) -> EmbeddingSuccessResponse

Parameters:

NameTypeRequiredDescription
input_typeCreateEmbeddingsRequestInputTypeYesThe type of content for the embeddings. Values: text, image, text_image, audio, video.
model_namestrYesThe video understanding model you wish to use. Value: marengo3.0.
textTextInputRequestNoText input configuration. Required when input_type is text. See TextInputRequest for details.
imageImageInputRequestNoImage input configuration. Required when input_type is image. See ImageInputRequest for details.
text_imageTextImageInputRequestNoCombined text and image input configuration. Required when input_type is text_image. See TextImageInputRequest for details.
audioAudioInputRequestNoAudio input configuration. Required when input_type is audio. See AudioInputRequest for details.
videoVideoInputRequestNoVideo input configuration. Required when input_type is video. See VideoInputRequest for details.
request_optionsRequestOptionsNoRequest-specific configuration.

TextInputRequest

The TextInputRequest class specifies configuration for processing text content. Required when input_type is text.

NameTypeRequiredDescription
input_textstrYesThe text for which you wish to create an embedding. The maximum length is 500 tokens.

ImageInputRequest

The ImageInputRequest class specifies configuration for processing image content. Required when input_type is image.

NameTypeRequiredDescription
media_sourceMediaSourceYesSpecifies the source of the image file. See MediaSource for details.

TextImageInputRequest

The TextImageInputRequest class specifies configuration for processing combined text and image content. Required when input_type is text_image.

NameTypeRequiredDescription
media_sourceMediaSourceYesSpecifies the source of the image file. See MediaSource for details.
input_textstrYesThe text for which you wish to create an embedding. The maximum length is 500 tokens.

AudioInputRequest

The AudioInputRequest class specifies configuration for processing audio content. Required when input_type is audio.

NameTypeRequiredDescription
media_sourceMediaSourceYesSpecifies the source of the audio file. See MediaSource for details.
start_secfloatNoThe start time in seconds for processing the audio file. Use this parameter to process a portion of the audio file starting from a specific time. Default: 0 (start from the beginning).
end_secfloatNoThe end time in seconds for processing the audio file. Use this parameter to process a portion of the audio file ending at a specific time. The end time must be greater than the start time. Default: End of the audio file.
segmentationAudioSegmentationNoSpecifies how the platform divides the audio into segments. When combined with embedding_scope=["clip"], creates separate embeddings for each segment. Use this to generate embeddings for specific portions of your audio. See AudioSegmentation for details.
embedding_optionList[str]NoThe types of embeddings you wish to generate. Values:
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech

You can specify multiple options to generate different types of embeddings for the same audio.
embedding_scopeList[str]NoThe scope for which you wish to generate embeddings. Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire audio file

You can specify multiple scopes to generate embeddings at different levels.

VideoInputRequest

The VideoInputRequest class specifies configuration for processing video content. Required when input_type is video.

NameTypeRequiredDescription
media_sourceMediaSourceYesSpecifies the source of the video file. See MediaSource for details.
start_secfloatNoThe start time in seconds for processing the video file. Use this parameter to process a portion of the video file starting from a specific time. Default: 0 (start from the beginning).
end_secfloatNoThe end time in seconds for processing the video file. Use this parameter to process a portion of the video file ending at a specific time. The end time must be greater than the start time. Default: End of the video file.
segmentationVideoSegmentationNoSpecifies how the platform divides the video into segments. When combined with embedding_scope=["clip"], creates separate embeddings for each segment. Supports fixed-duration segments or dynamic segmentation that adapts to scene changes. See VideoSegmentation for details.
embedding_optionList[str]NoThe types of embeddings to generate for the video. Values:
- visual: Generates embeddings based on visual content (scenes, objects, actions)
- audio: Generates embeddings based on audio content (sounds, music, effects)
- transcription: Generates embeddings based on transcribed speech

You can specify multiple options to generate different types of embeddings for the same video. Default: ["visual", "audio", "transcription"].
embedding_scopeList[str]NoThe scope for which you wish to generate embeddings. Values:
- clip: Generates one embedding for each segment
- asset: Generates one embedding for the entire video file

You can specify multiple scopes to generate embeddings at different levels. Default: ["clip", "asset"].

MediaSource

The MediaSource class specifies the source of the media file. Provide exactly one of the following:

NameTypeRequiredDescription
base64_stringstrNoThe base64-encoded media data.
urlstrNoThe publicly accessible URL of the media file. Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported.
asset_idstrNoThe unique identifier of an asset from a direct or multipart upload.

AudioSegmentation

The AudioSegmentation class specifies how the platform divides the audio into segments using fixed-length intervals.

NameTypeRequiredDescription
strategyLiteral["fixed"]YesThe segmentation strategy. Value: fixed.
fixedAudioSegmentationFixedYesConfiguration for fixed segmentation. See AudioSegmentationFixed for details.

AudioSegmentationFixed

The AudioSegmentationFixed class configures fixed-length segmentation for audio.

NameTypeRequiredDescription
duration_secintYesThe duration in seconds for each segment. The platform divides the audio into segments of this exact length. The final segment may be shorter if the audio duration is not evenly divisible.

Example: With duration_sec: 5, a 12-second audio file produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentation

The VideoSegmentation type specifies how the platform divides the video into segments. Use one of the following:

Fixed segmentation: Divides the video into equal-length segments:

NameTypeRequiredDescription
strategyLiteral["fixed"]YesThe segmentation strategy. Value: fixed.
fixedVideoSegmentationFixedFixedYesConfiguration for fixed segmentation. See VideoSegmentationFixedFixed for details.

Dynamic segmentation: Divides the video into adaptive segments based on scene changes:

NameTypeRequiredDescription
strategyLiteral["dynamic"]YesThe segmentation strategy. Value: dynamic.
dynamicVideoSegmentationDynamicDynamicYesConfiguration for dynamic segmentation. See VideoSegmentationDynamicDynamic for details.

VideoSegmentationFixedFixed

The VideoSegmentationFixedFixed class configures fixed-length segmentation for video.

NameTypeRequiredDescription
duration_secintYesThe duration in seconds for each segment. The platform divides the video into segments of this exact length. The final segment may be shorter if the video duration is not evenly divisible.

Example: With duration_sec: 5, a 12-second video produces segments: [0-5s], [5-10s], [10-12s].

VideoSegmentationDynamicDynamic

The VideoSegmentationDynamicDynamic class configures dynamic segmentation for video based on scene changes.

NameTypeRequiredDescription
min_duration_secintYesThe minimum duration in seconds for each segment. The platform divides the video into segments that are at least this long. Segments adapt to scene changes and content boundaries and may be longer than the minimum.

Example: With min_duration_sec: 3, segments might be: [0-3.2s], [3.2-7.8s], [7.8-12.1s].

Return value: Returns an EmbeddingSuccessResponse object containing the embedding results.

The EmbeddingSuccessResponse class contains the following properties:

NameTypeDescription
dataList[EmbeddingData]Array of embedding results.
metadataOptional[EmbeddingMediaMetadata]Metadata about the media content.

The EmbeddingData class contains the following properties:

NameTypeDescription
embeddingList[float]The embedding vector for the content.
embedding_optionOptional[EmbeddingDataEmbeddingOption]The type of embedding. Values: visual, audio, transcription.
embedding_scopeOptional[EmbeddingDataEmbeddingScope]The scope of the embedding. Values: clip, asset.
start_secOptional[float]The start time in seconds for this embedding segment.
end_secOptional[float]The end time in seconds for this embedding segment.

API Reference: Create sync embeddings