Make any-to-video search requests

Use this endpoint to search for relevant matches in an index using text, media, or a combination of both as your query.

Text queries:

Use the query_text parameter to specify your query.

Media queries:

Set the query_media_type parameter to the corresponding media type (example: image).
Provide up to 10 images by specifying the following parameters multiple times:
- query_media_url: Publicly accessible URL of your media file.
- query_media_file: Local media file. Composed text and media queries:
Use the query_text parameter for your text query.
Set query_media_type to image.
Provide up to 10 images by specifying the query_media_url and query_media_file parameters multiple times.

Entity search (beta):

To find a specific person in your videos, enclose the unique identifier of the entity you want to find in the query_text parameter.

Notes

When using images in your search queries (either as media queries or in composed searches), ensure your image files meet the requirements.
This endpoint is rate-limited. For details, see the Rate limits page.

Use this endpoint to search for relevant matches in an index using text, media, or a combination of both as your query. **Text queries**: - Use the `query_text` parameter to specify your query. **Media queries**: - Set the `query_media_type` parameter to the corresponding media type (example: `image`). - Provide up to 10 images by specifying the following parameters multiple times: - `query_media_url`: Publicly accessible URL of your media file. - `query_media_file`: Local media file. **Composed text and media queries**: - Use the `query_text` parameter for your text query. - Set `query_media_type` to `image`. - Provide up to 10 images by specifying the `query_media_url` and `query_media_file` parameters multiple times. **Entity search** (beta): - To find a specific person in your videos, enclose the unique identifier of the entity you want to find in the `query_text` parameter. <Note title="Notes"> - When using images in your search queries (either as media queries or in composed searches), ensure your image files meet the [requirements](/v1.3/docs/concepts/models/marengo#image-file-requirements). - This endpoint is rate-limited. For details, see the [Rate limits](/v1.3/docs/get-started/rate-limits) page. </Note>

Authentication

x-api-keystring

Your API key.

Note

You can find your API key on the API Keys page.

Your API key. <Note title="Note"> You can find your API key on the <a href="https://playground.twelvelabs.io/dashboard/api-keys" target="_blank">API Keys</a> page. </Note>

Request

Request to perform a search on a video index.

query_media_typeenumOptional

The type of media you wish to use. This parameter is required for media queries. For example, to perform an image-based search, set this parameter to image. Use query_text together with this parameter when you want to perform a composed image+text search.

Allowed values:

query_media_urlstring or list of stringsOptional

The publicly accessible URL of a media file to use as a query. This parameter is required for media queries if query_media_file is not provided.

You can provide up to 10 images by specifying this parameter multiple times:

--form query_media_url=https://example.com/image1.jpg \
--form query_media_url=https://example.com/image2.jpg

The publicly accessible URL of a media file to use as a query. This parameter is required for media queries if `query_media_file` is not provided. You can provide up to 10 images by specifying this parameter multiple times: ``` --form query_media_url=https://example.com/image1.jpg \ --form query_media_url=https://example.com/image2.jpg ```

query_media_filestring or list of stringsOptional

A local media file to use as a query. This parameter is required for media queries if query_media_url is not provided.

You can provide up to 10 images by specifying this parameter multiple times:

--form query_media_file=@/path/to/image1.jpg \
--form query_media_file=@/path/to/image2.jpg

A local media file to use as a query. This parameter is required for media queries if `query_media_url` is not provided. You can provide up to 10 images by specifying this parameter multiple times: ``` --form query_media_file=@/path/to/image1.jpg \ --form query_media_file=@/path/to/image2.jpg ```

query_textstringOptional

If you’re using the Entity Search feature to search for specific persons in your video content, you must enclose the unique identifier of your entity between the <@ and > markers. For example, to search for an entity with the ID entity123, use <@entity123> is walking as your query.

Marengo supports up to 500 tokens per query.

The text query to search for. This parameter is required for text queries. Note that the platform supports full natural language-based search. You can use this parameter together with `query_media_type` and `query_media_url` or `query_media_file` to perform a composed image+text search. If you're using the Entity Search feature to search for specific persons in your video content, you must enclose the unique identifier of your entity between the `<@` and `>` markers. For example, to search for an entity with the ID `entity123`, use `<@entity123> is walking` as your query. Marengo supports up to 500 tokens per query.

index_idstringRequired

The unique identifier of the index to search.

search_optionslist of enumsRequired

Specifies the modalities the video understanding model uses to find relevant information.

Available options:

visual: Searches visual content.
audio: Searches non-speech audio.
transcription: Spoken words

Note

You can specify multiple search options in conjunction with the operator parameter described below to broaden or narrow your search. For example, to search using both visual and non-speech audio content, include this parameter two times in the request as shown below:

1 --form search_options=visual \
2 --form search_options=audio \
3 --form search_options=transcription \

For guidance, see the Search options section.

Specifies the modalities the video understanding model uses to find relevant information. Available options: - `visual`: Searches visual content. - `audio`: Searches non-speech audio. - `transcription`: Spoken words <Note title="Note"> - You can specify multiple search options in conjunction with the [`operator`](/v1.3/api-reference/any-to-video-search/make-search-request#request.body.operator.operator) parameter described below to broaden or narrow your search. For example, to search using both visual and non-speech audio content, include this parameter two times in the request as shown below: ```JSON --form search_options=visual \ --form search_options=audio \ --form search_options=transcription \ ``` </Note> For guidance, see the [Search options](/v1.3/docs/concepts/modalities#search-options) section.

Allowed values:

transcription_optionslist of enumsOptional

Specifies how the platform matches your text query with the words spoken in the video. This parameter applies only when the search_options parameter contains the transcription value.

Available options:

lexical: Exact word matching
semantic: Meaning-based matching

For details on when to use each option, see the Transcription options section.

Default: ["lexical", "semantic"].

Specifies how the platform matches your text query with the words spoken in the video. This parameter applies only when the `search_options` parameter contains the `transcription` value. Available options: - `lexical`: Exact word matching - `semantic`: Meaning-based matching For details on when to use each option, see the [Transcription options](/v1.3/docs/concepts/modalities#transcription-options) section. **Default**: `["lexical", "semantic"]`.

Allowed values:

group_byenumOptionalDefaults to clip

Use this parameter to group or ungroup items in a response. It can take one of the following values:

video: The platform will group the matching video clips in the response by video.
clip: The matching video clips in the response will not be grouped.

Default: clip

Use this parameter to group or ungroup items in a response. It can take one of the following values: - `video`: The platform will group the matching video clips in the response by video. - `clip`: The matching video clips in the response will not be grouped. **Default:** `clip`

Allowed values:

operatorenumOptionalDefaults to or

Combines multiple search options using or or and. Use and to find segments matching all search options. Use or to find segments matching any search option. For detailed guidance on using this parameter, see the Combine multiple modalities section.

Default: or.

Combines multiple search options using `or` or `and`. Use `and` to find segments matching all search options. Use `or` to find segments matching any search option. For detailed guidance on using this parameter, see the [Combine multiple modalities](/v1.3/docs/concepts/modalities#combine-multiple-modalities) section. **Default**: `or`.

Allowed values:

page_limitintegerOptionalDefaults to 10

The number of items to return on each page. When grouping by video, this parameter represents the number of videos per page. Otherwise, it represents the maximum number of video clips per page.

Max: 50.

filterstringOptional

Specifies a stringified JSON object to filter your search results. Supports both system-generated metadata (example: video ID, duration) and user-defined metadata.

Syntax for filtering

The following table describes the supported data types, operators, and filter syntax:

Data type	Operator	Description	Syntax
String	`=`	Matches results equal to the specified value.	`{"field": "value"}`
Array of strings	`=`	Matches results with any value in the specified array. Supported only for `id`.	`{"id": ["value1", "value2"]}`
Numeric (integer, float)	`=`, `lte`, `gte`	Matches results equal to or within a range of the specified value.	`{"field": number}` or `{"field": { "gte": number, "lte": number }}`
Boolean	`=`	Matches results equal to the specified boolean value.	`{"field": true}` or `{"field": false}`.

System-generated metadata

The table below describes the system-generated metadata available for filtering your search results:

Field name	Description	Type	Example
`id`	Filters by specific video IDs.	Array of strings	`{"id": ["67cec9caf45d9b64a58340fc", "67cec9baf45d9b64a58340fa"]}`.
`duration`	Filters based on the duration of the video containing the segment that matches your query.	Number or object with `gte` and `lte`	`{"duration": 600}` or `{"duration": { "gte": 600, "lte": 800 }}`
`width`	Filters by video width (in pixels).	Number or object with `gte` and `lte`	`{"width": 1920}` or `{"width": { "gte": 1280, "lte": 1920}}`
`height`	Filters by video height (in pixels).	Number or object with `gte` and `lte`.	`{"height": 1080}` or `{"height": { "gte": 720, "lte": 1080 }}`.
`size`	Filters by video size (in bytes)	Number or object with `gte` and `lte`.	`{"size": 1048576}` or `{"size": { "gte": 1048576, "lte": 5242880}}`
`filename`	Filters by the exact file name.	String	`{"filename": "Animal Encounters part 1"}`

User-defined metadata

To filter by user-defined metadata:

Add metadata to your video by calling the PUT method of the /indexes/:index-id/videos/:video-id endpoint
Reference the custom field in your filter object. For example, to filter videos where a custom field named needsReview of type boolean is true, use {"needs_review": true}.

For more details and examples, see the Filter search results page.

Specifies a stringified JSON object to filter your search results. Supports both system-generated metadata (example: video ID, duration) and user-defined metadata. **Syntax for filtering** The following table describes the supported data types, operators, and filter syntax: | Data type | Operator | Description | Syntax | |:----------|:---------|:------------|:-------| | String | `=` | Matches results equal to the specified value. | `{"field": "value"}` | Array of strings | `=` | Matches results with any value in the specified array. Supported only for `id`. | `{"id": ["value1", "value2"]}` | | Numeric (integer, float) | `=`, `lte`, `gte` | Matches results equal to or within a range of the specified value. | `{"field": number}` or `{"field": { "gte": number, "lte": number }}` | | Boolean | `=` | Matches results equal to the specified boolean value. | `{"field": true}` or `{"field": false}`. | <br/> **System-generated metadata** The table below describes the system-generated metadata available for filtering your search results: | Field name | Description | Type | Example | |:-----------|:------------|:-----|:--------| | `id` | Filters by specific video IDs. | Array of strings | `{"id": ["67cec9caf45d9b64a58340fc", "67cec9baf45d9b64a58340fa"]}`. | | `duration` | Filters based on the duration of the video containing the segment that matches your query. | Number or object with `gte` and `lte` | `{"duration": 600}` or `{"duration": { "gte": 600, "lte": 800 }}` | | `width` | Filters by video width (in pixels). | Number or object with `gte` and `lte` | `{"width": 1920}` or `{"width": { "gte": 1280, "lte": 1920}}` | | `height` | Filters by video height (in pixels). | Number or object with `gte` and `lte`. | `{"height": 1080}` or `{"height": { "gte": 720, "lte": 1080 }}`. | | `size` | Filters by video size (in bytes) | Number or object with `gte` and `lte`. | `{"size": 1048576}` or `{"size": { "gte": 1048576, "lte": 5242880}}` | | `filename` | Filters by the exact file name. | String | `{"filename": "Animal Encounters part 1"}` | <br/> **User-defined metadata** To filter by user-defined metadata: 1. Add metadata to your video by calling the [`PUT`](/v1.3/api-reference/videos/update) method of the `/indexes/:index-id/videos/:video-id` endpoint 2. Reference the custom field in your filter object. For example, to filter videos where a custom field named `needsReview` of type boolean is `true`, use `{"needs_review": true}`. For more details and examples, see the [Filter search results](/v1.3/docs/guides/search/filtering) page.

include_user_metadatabooleanOptional

Specifies whether to include user-defined metadata in the search results.

Response

Successfully performed a search request.

datalist of objects

An array that contains your search results. For each match found, the model returns the following fields:

page_infoobject

An object that provides information about pagination.

search_poolobject

An object that contains details about the index you queried.

Errors

400

Bad Request Error

429

Too Many Requests Error

Specifies a stringified JSON object to filter your search results. Supports both system-generated metadata (example: video ID, duration) and user-defined metadata.

Syntax for filtering

The following table describes the supported data types, operators, and filter syntax:

Data type	Operator	Description	Syntax
String	`=`	Matches results equal to the specified value.	`{"field": "value"}`
Array of strings	`=`	Matches results with any value in the specified array. Supported only for `id`.	`{"id": ["value1", "value2"]}`
Numeric (integer, float)	`=`, `lte`, `gte`	Matches results equal to or within a range of the specified value.	`{"field": number}` or `{"field": { "gte": number, "lte": number }}`
Boolean	`=`	Matches results equal to the specified boolean value.	`{"field": true}` or `{"field": false}`.

System-generated metadata

The table below describes the system-generated metadata available for filtering your search results:

Field name	Description	Type	Example
`id`	Filters by specific video IDs.	Array of strings	`{"id": ["67cec9caf45d9b64a58340fc", "67cec9baf45d9b64a58340fa"]}`.
`duration`	Filters based on the duration of the video containing the segment that matches your query.	Number or object with `gte` and `lte`	`{"duration": 600}` or `{"duration": { "gte": 600, "lte": 800 }}`
`width`	Filters by video width (in pixels).	Number or object with `gte` and `lte`	`{"width": 1920}` or `{"width": { "gte": 1280, "lte": 1920}}`
`height`	Filters by video height (in pixels).	Number or object with `gte` and `lte`.	`{"height": 1080}` or `{"height": { "gte": 720, "lte": 1080 }}`.
`size`	Filters by video size (in bytes)	Number or object with `gte` and `lte`.	`{"size": 1048576}` or `{"size": { "gte": 1048576, "lte": 5242880}}`
`filename`	Filters by the exact file name.	String	`{"filename": "Animal Encounters part 1"}`

User-defined metadata

To filter by user-defined metadata:

Add metadata to your video by calling the PUT method of the /indexes/:index-id/videos/:video-id endpoint
Reference the custom field in your filter object. For example, to filter videos where a custom field named needsReview of type boolean is true, use {"needs_review": true}.

For more details and examples, see the Filter search results page.

1	curl -X POST https://api.twelvelabs.io/v1.3/search \
2	-H "x-api-key: <apiKey>" \
3	-H "Content-Type: multipart/form-data" \
4	-F query_text="A man walking a dog" \
5	-F index_id="6298d673f1090f1100476d4c" \
6	-F search_options="visual" \
7	-F group_by="clip" \
8	-F operator="or" \
9	-F page_limit='10' \
10	-F filter="{\"id\":[\"66284191ea717fa66a274832\"]}"

1	{
2	"data": [
3	{
4	"start": 238.75,
5	"end": 259.62109375,
6	"video_id": "639963a1ce36463e0199c8c7",
7	"rank": 1,
8	"thumbnail_url": "https://example.com/thumbnail.jpg",
9	"transcription": "A woman vlogs about her summer day, sharing her experience"
10	}
11	],
12	"search_pool": {
13	"total_count": 10,
14	"total_duration": 8731,
15	"index_id": "639961c9e219c90227c371a2"
16	}
17	}

1	--form search_options=visual \
2	--form search_options=audio \
3	--form search_options=transcription \