Create embeddings for text, image, and audio

This method creates embeddings for text, image, and audio content.

Before you create an embedding, ensure that your image or audio files meet the following prerequisites:

Parameters for embeddings:

Common parameters:
- model_name: The video understanding model you want to use. Example: “Marengo-retrieval-2.7”.
Text embeddings:
- text: Text for which to create an embedding.
Image embeddings: Provide one of the following:
- image_url: Publicly accessible URL of your image file.
- image_file: Local image file.
Audio embeddings: Provide one of the following:
- audio_url: Publicly accessible URL of your audio file.
- audio_file: Local audio file.

Notes

The Marengo video understanding model generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content.
You can create multiple types of embeddings in a single API call.
Audio embeddings combine generic sound and human speech in a single embedding. For videos with transcriptions, you can retrieve transcriptions and then create text embeddings from these transcriptions.

This method creates embeddings for text, image, and audio content. Before you create an embedding, ensure that your image or audio files meet the following prerequisites: - [Image embeddings](/v1.3/docs/guides/create-embeddings/image#prerequisites) - [Audio embeddings](/v1.3/docs/guides/create-embeddings/audio#prerequisites) Parameters for embeddings: - **Common parameters**: - `model_name`: The video understanding model you want to use. Example: "Marengo-retrieval-2.7". - **Text embeddings**: - `text`: Text for which to create an embedding. - **Image embeddings**: Provide one of the following: - `image_url`: Publicly accessible URL of your image file. - `image_file`: Local image file. - **Audio embeddings**: Provide one of the following: - `audio_url`: Publicly accessible URL of your audio file. - `audio_file`: Local audio file. <Note title="Notes"> - The Marengo video understanding model generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content. - You can create multiple types of embeddings in a single API call. - Audio embeddings combine generic sound and human speech in a single embedding. For videos with transcriptions, you can retrieve transcriptions and then [create text embeddings](/v1.3/api-reference/text-image-audio-embeddings/create-text-image-audio-embeddings) from these transcriptions. </Note>

Headers

x-api-keystringRequired

Request

Request to create an embedding synchronously.

model_namestringRequired

The name of the model you want to use. The following models are available:

Marengo-retrieval-2.7

textstringOptional

The text for which you wish to create an embedding.

Note

Text embeddings are limited to 77 tokens. If the text exceeds this limit, the platform truncates it according to the value of the text_truncate parameter described below.

Example: “Man with a dog crossing the street”

The text for which you wish to create an embedding. <Note title="Note"> Text embeddings are limited to 77 tokens. If the text exceeds this limit, the platform truncates it according to the value of the `text_truncate` parameter described below. </Note> **Example**: "Man with a dog crossing the street"

text_truncatestringOptionalDefaults to end

Specifies how the platform truncates text that exceeds 77 tokens to fit the maximum length allowed for an embedding. This parameter can take one of the following values:

start: The platform will truncate the start of the provided text.
end: The platform will truncate the end of the provided text.
none: The platform will return an error if the text is longer than the maximum token limit.

Default: end

Specifies how the platform truncates text that exceeds 77 tokens to fit the maximum length allowed for an embedding. This parameter can take one of the following values: - `start`: The platform will truncate the start of the provided text. - `end`: The platform will truncate the end of the provided text. - `none`: The platform will return an error if the text is longer than the maximum token limit. **Default**: `end`

image_urlstringOptionalformat: "uri"

The publicly accessible URL of the image for which you wish to create an embedding. This parameter is required for image embeddings if image_file is not provided.

image_filefileOptional

The image file for which you wish to create an embedding as a local file. This parameter is required for image embeddings if image_url is not provided.

audio_urlstringOptionalformat: "uri"

The publicly accessible URL of the audio file for which you wish to create an embedding. This parameter is required for audio embeddings if audio_file is not provided.

audio_filefileOptional

The audio file for which you wish to create an embedding as a local file. This parameter is required for audio embeddings if audio_url is not provided.

audio_start_offset_secdoubleOptionalDefaults to 0

Specifies the start time, in seconds, from which the platform generates the audio embeddings. This parameter allows you to skip the initial portion of the audio during processing. Default: 0.

Response

A text embedding has successfully been created.

model_namestring

The name of the video understanding model the platform has used to create this embedding.

text_embeddingobject or null

An object that contains the generated text embedding vector and associated information. Present when text was processed.

image_embeddingobject or null

An object that contains the generated image embedding vector and associated information. Present when image was processed.

audio_embeddingobject or null

An object that contains the generated audio embedding vector and associated information. Present when audio was processed.

Errors

This method creates embeddings for text, image, and audio content.

Before you create an embedding, ensure that your image or audio files meet the following prerequisites:

Parameters for embeddings:

Common parameters:
- model_name: The video understanding model you want to use. Example: “Marengo-retrieval-2.7”.
Text embeddings:
- text: Text for which to create an embedding.
Image embeddings: Provide one of the following:
- image_url: Publicly accessible URL of your image file.
- image_file: Local image file.
Audio embeddings: Provide one of the following:
- audio_url: Publicly accessible URL of your audio file.
- audio_file: Local audio file.

Notes

The Marengo video understanding model generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content.
You can create multiple types of embeddings in a single API call.
Audio embeddings combine generic sound and human speech in a single embedding. For videos with transcriptions, you can retrieve transcriptions and then create text embeddings from these transcriptions.

1	curl -X POST https://api.twelvelabs.io/v1.3/embed \
2	-H "x-api-key: <apiKey>" \
3	-H "Content-Type: multipart/form-data" \
4	-F model_name="Marengo-retrieval-2.7" \
5	-F text="Man with a dog crossing the street" \
6	-F image_file=@<file1> \
7	-F audio_file=@<file1>

1	{
2	"model_name": "Marengo-retrieval-2.7",
3	"text_embedding": {
4	"segments": [
5	{
6	"float": [
7	-0.042632885,
8	0.014312328,
9	0.029209055
10	]
11	}
12	]
13	},
14	"image_embedding": {
15	"segments": [
16	{
17	"float": [
18	-0.042632885,
19	0.014312328,
20	0.029209055
21	]
22	}
23	],
24	"metadata": {
25	"input_filename": "01.jpg"
26	}
27	},
28	"audio_embedding": {
29	"segments": [
30	{
31	"float": [
32	-0.042632885,
33	0.014312328,
34	0.029209055
35	],
36	"start_offset_sec": 0
37	}
38	],
39	"metadata": {
40	"input_filename": "01.mp3"
41	}
42	}
43	}