Provide an image, and the platform finds all the semantically related video segments where similar content appears.

This feature allows you to find videos containing objects, colors, and shapes similar to those in the images you provide. Key use cases include:

Online shopping: Easily search for products similar but not identical to the one you like.
Creative projects: Find clips with buildings, scenery, or objects that match a reference image.
Visual search: Find relevant videos when the desired results are challenging to describe with text.

Note the following about using images as queries:

The platform supports only semantic searches. When performing a semantic search, the platform determines the meaning of the image you provide and finds the video segments containing contextually similar elements. For example, if you use an image of a tree as the query, the search results might contain videos featuring different trees, focusing on the overall characteristics rather than specific details.
The objects you want to search for must be sufficiently large and detailed. For example, using an image of a car in a parking lot is more likely to yield precise results than an image of a small branded pen on a table in a large room.
The platform does not support searching for specific words or phrases spoken or displayed as text within videos. For example, if you provide an image of a cat as your query and want to find all the video segments where the word "cat" is mentioned or appears on the screen, image queries cannot retrieve those results. Use text queries instead.

Examples

This section provides examples of performing search requests using image queries. You can provide images as local files or publicly accessible URLs. Use the query_media_file parameter for local image files and the query_media_url parameter for publicly accessible URLs.

Ensure that the prerequisites are met before proceeding.

Visual

The following example code searches within the "visual" modality, excluding spoken words. It invokes the query method of the search object with the following parameters:

index_id: The unique identifier of the index containing your videos.
query_media_type: The type of query. This example uses image.
query_media_file: The path of the image file you wish to provide.
options: An array containing the source of information the platform must use. This example uses ["visual"]

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query_media_type="image",
    query_media_file="<YOUR_FILE_PATH>", # Use query_media_url instead to provide a file from a publicly accessible URL.
    options=["visual"]
)

# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence} metadata={clip.metadata}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  queryMediaType: 'image',
  queryMediaFile: '<YOUR_FILE_PATH>', // Use queryMediaUrl instead to provide a file from a publicly accessible URL.
  options: ['visual']
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData: any) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence} metadata=${JSON.stringify(clip.metadata)}`,
    );
  });
}

The example output below was truncated for brevity:

video_id=65d60bcf48db9fa780cb415e score=83.73 start=273.96875 end=289.0625 confidence=high metadata=[{'type': 'visual'}]
video_id=65d60bcf48db9fa780cb415e score=83.55 start=397.921875 end=439.84375 confidence=high metadata=[{'type': 'visual'}]
video_id=65d60bcf48db9fa780cb415e score=83.46 start=294.5625 end=311.84375 confidence=high metadata=[{'type': 'visual'}]
video_id=65d60bcf48db9fa780cb415e score=83.45 start=233.0 end=247.78125 confidence=high metadata=[{'type': 'visual'}]

Visual and logo

The following example code searches within the "visual" and "logo" modalities. It invokes the query method of the search object with the following parameters:

index_id: The unique identifier of the index containing your videos.
query_media_type: The type of query. This example uses image.
query_media_file: The path of the image file you wish to provide.
options: An array containing the source of information the platform must use. This example uses ["visual", "logo"]

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query_media_type="image",
    query_media_file="<YOUR_FILE_PATH>",# Use query_media_url instead to provide a file from a publicly accessible URL
    options=["visual", "logo"]
)

# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence} metadata={clip.metadata}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  queryMediaType: 'image',
  queryMediaFile: '<YOUR_FILE_PATH>', // Use queryMediaUrl instead to provide a file from a publicly accessible URL.
  options: ['visual', 'logo']
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData: any) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence} metadata=${JSON.stringify(clip.metadata)}`,
    );
  });
}

The example output below was truncated for brevity:

video_id=65d60bcf48db9fa780cb415e score=83.73 start=273.96875 end=289.0625 confidence=high metadata=[{'type': 'visual'}]
video_id=63eb67e49678ed7709731d63 score=92.28 start=104 end=106 confidence=high metadata=[{'type': 'logo', 'text': 'Starbucks'}]