Videos contain combinations of images, spoken words, sounds, movements, and text. When you have a large library of videos, it might be difficult to know what's in your videos and how to write a query that finds the exact moments you need. Although the platform doesn't automatically extract any tags from your videos and doesn't require you to use them when you perform searches, extracting information that appears in a video can help you get insights into the content of your videos. Instead of watching the entire video and annotating it manually, you can use information extracted by the platform to perform more fine-grained searches.

Prerequisites

You're familiar with the concepts that are described on the Platform overview page.
You've already created an index, and the Marengo video understanding engine is enabled for this index.
You've uploaded a video, and the platform has finished indexing it.
To extract logos, the logo indexing option must be enabled for the index containing your video.
To extract thumbnails, the thumbnail add-on must be enabled for the index containing your video.

Extract transcriptions

To extract transcriptions, call the transcription method of the index.video object with the following parameters:

index_id: A string representing the unique identifier of the index containing the video for which you want to extract transcriptions.
video_id: A string representing the unique identifier of the video.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

transcriptions = client.index.video.transcription(
    index_id="<YOUR_INDEX_ID>",
    id="<YOUR_VIDEO_ID>"
)

for transcription in transcriptions:
    print(
        f"value={transcription.value} start={transcription.start} end={transcription.end}"
    )

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

const transcriptions = await client.index.video.transcription('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>');
transcriptions.forEach((transcription) => {
  console.log(`  value=${transcription.value} start=${transcription.start} end=${transcription.end}`);
});

The following example output was truncated for brevity:

value=And it's super economical if you don't want to buy a bunch of mechanical things like a food processor. start=78.484 end=82.333
value=Next thing is a personal favorite of mine, start=82.344 end=83.874
value=which is a pair of nice kitchen shears. start=83.958 end=85.858
value=These come in handy way more than you would think. start=85.867 end=88.05

Note that you can use the start and end query parameters to specify the time range for which you want to retrieve the transcription. The following example retrieves the transcription for the first 10 seconds of the video:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

transcriptions = client.index.video.transcription(
    index_id="<YOUR_INDEX_ID>",
    id="<YOUR_VIDEO_ID>",
    start=0,
    end=10
)

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

const transcriptions = await client.index.video.transcription('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>', { start: 0, end: 10 });
transcriptions.forEach((transcription) => {
  console.log(`value=${transcription.value} start=${transcription.start} end=${transcription.end}`);
});

Extract text recognized in a video(OCR)

To extract the text recognized in a video, call the text_in_video method of the index.video object with the following parameters:

index_id: A string representing the unique identifier of the index containing the video for which you extract text.
video_id: A string representing the unique identifier of the video.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

texts = client.index.video.text_in_video(
    index_id="<YOUR_INDEX_ID>",
    id="<YOUR_VIDEO_ID>"
)

for text in texts:
    print(
        f"value={text.value} start={text.start} end={text.end}"
    )

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

const texts = await client.index.video.textInVideo('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>');
texts.forEach((text) => {
  console.log(`value=${text.value} start=${text.start} end=${text.end}`);
});

Note that you can use the start and end parameters to specify the time range for which you want to retrieve the text. The following example retrieves the text for the first 10 seconds of the video:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

texts = client.index.video.text_in_video(
    index_id=INDEX_ID,
    id=VIDEO_ID
    start=0,
    end=10
  )

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

const texts = await client.index.video.textInVideo('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>', { start: 0, end: 10 });
texts.forEach((text) => {
  console.log(`value=${text.value} start=${text.start} end=${text.end}`);
});

Extract logos

To extract logos, call the logo method of the index.video object with the following parameters:

index_id: A string representing the unique identifier of the index containing the video for which you want to extract logos.
video_id: A string representing the unique identifier of the video.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

logos = client.index.video.logo(
    index_id=INDEX_ID,
    id=VIDEO_ID
  )

for logo in logos:
    print(
        f"value={logo.value} start={logo.start} end={logo.end}"
    )

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

const logos = await client.index.video.logo('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>');
logos.forEach((logo) => {
  console.log(`value=${logo.value} start=${logo.start} end=${logo.end}`);
});

The output should look similar to the following one:

value=Hyundai Motor Company start=167 end=168
value=Hyundai Motor Company start=224 end=226

In this example output, the Hyundai logo appears in the specified video twice, as shown below:

At 167 seconds from the start of the video:
At 224 seconds from the start of the video:

Note that you can use the start and end parameters to specify the time range for which you want to extract logos. The following example extracts logos for the first 10 seconds of the video:

logos = client.index.video.logo(
    index_id="<YOUR_INDEX_ID>",
    id="<YOUR_VIDEO_ID>,
    start=0
    end=10
  )

const logos = await client.index.video.logo('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID', { start: 0, end: 10 });
logos.forEach((logo) => {
  console.log(`value=${logo.value} start=${logo.start} end=${logo.end}`);
});

Retrieve thumbnails

The platform allows you to retrieve thumbnails for your videos or search results that you can display in your application. For example, you can retrieve thumbnails when:

Your application displays previews for one or more videos. For example, users may want to see thumbnails of all the videos in an index before deciding which video they wish to delete.
Your application displays previews for search results. For example, when users perform a search, your application could use thumbnails to give users a small view of each matching video clip.

Note the following about retrieving thumbnails:

For videos: The platform allows you to retrieve a thumbnail from the middle of the video or a specific time, measured in seconds, from the beginning.
For search results: The platform always returns a thumbnail from the middle of each matching video clip.

Each URL is unique and remains valid for one hour.

Depending on your use case, refer to the appropriate section below.

Retrieve a thumbnail of a video

To retrieve a thumbnail from the middle of a video, call the thumbnail method of the index.video object with the following parameters:

index_id: A string representing the unique identifier of the index containing the video for which you want to retrieve a thumbnail.
video_id: A string representing the unique identifier of the video.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

thumbnail = client.index.video.thumbnail(
  index_id="<YOUR_INDEX_ID>",
  id="<YOUR_VIDEO_ID>"
)
print(f"Thumbnail URL: {thumbnail}")

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

const thumbnail = await client.index.video.thumbnail('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>'>);
console.log(`Thumbnail URL: ${thumbnail}`);

The output should look similar to the following one:

Thumbnail URL: https://project-one-thumbnail.s3.us-west-2.amazonaws.com/65b9f6cd4981af6e637c9537/231.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T074755Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=deefd2dd20c576bf91efa3249f3c5bb5061efe09f5b64c10b92c964988201d0e

The following example code uses the time parameter to specify the time, in seconds, at which the platform must retrieve a thumbnail:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

thumbnail = client.index.video.thumbnail(
  index_id="<YOUR_INDEX_ID>",
  id="<YOUR_VIDEO_ID>"
  time=10
  )
print(f"Thumbnail URL: {thumbnail}")

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});
const thumbnail = await client.index.video.thumbnail('<YOUR_INDEX_ID>', '<YOUR_VIDEO_ID>', 10);
console.log(`Thumbnail URL: ${thumbnail}`);

Retrieve thumbnails for search results

When you perform a search on an index for which thumbnail generation is enabled, the platform returns a field named thumbnail_url that contains a thumbnail from the middle of each matching video clip.

The following example code performs a search on an index for which thumbnail generation is enabled. Make sure to replace the placeholders surrounded by <> with your values:

import { TwelveLabs } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>",
  query= {
    "text": "<YOUR_QUERY>"
  },
  options=["visual"],
)
# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f" video_id={clip.video_id} thumbnail={clip.thumbnail_url} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>'
  query: '<YOUR_QUERY>',
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} thumbnail=${clip.thumbnailUrl} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence} metadata=${JSON.stringify(clip.metadata)}`,
    );
  });
}

The following example output was truncated for brevity:

video_id=65e013b448db9fa780cb4379 thumbnail=https://project-one-thumbnail.s3.us-west-2.amazonaws.com/65e013b448db9fa780cb4379/18.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20240229%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240229T053220Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=4af04ecea996ab63ecdaa6e06dbc024f59eab6e2443189de847c477ff08786df score=53.02 start=17.0 end=22.0 confidence=low
video_id=65e013b448db9fa780cb4379 thumbnail=https://project-one-thumbnail.s3.us-west-2.amazonaws.com/65e013b448db9fa780cb4379/416.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20240229%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240229T053220Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=b05a18f98979ba0a2be639f384bc4086853ef2559e9f316eed3fb16e0a9fff04 score=48.3 start=415.0 end=419.0 confidence=low
video_id=65e013b448db9fa780cb4379 thumbnail=https://project-one-thumbnail.s3.us-west-2.amazonaws.com/65e013b448db9fa780cb4379/671.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20240229%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240229T053220Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=501c6ae55de84bac79fa837bf8c4124e2c8cbf86d2e6b6a717d1deea3f8acf4d score=48.04 start=670.0 end=674.0 confidence=low