This guide shows how you can classify the content of all the videos within an index.

Prerequisites

You’re familiar with the concepts that are described on the Platform overview page.
You've already created an index, and the Marengo video understanding engine is enabled for this index.
You've uploaded one or more videos, and the platform has finished indexing them. For details, see the Upload videos page.
The unique identifier of the index to which you've uploaded your videos is stored in a variable named INDEX_ID.

Examples

This section shows several examples of classifying videos and specifying various parameters. For a description of each field in the request and response, see the API Reference > Classify all the videos in an index page.

Classify based on visual cues

To classify all the videos within an index based on visual cues, invoke the index method of the classify object with the following parameters:

index_id: A string representing the unique identifier of the index containing the videos you wish to classify.
classes: An array of objects containing the names and definitions of the entities or actions that the platform must identify.
options: An array of strings specifying the sources of information the platform uses when categorizing a video. For details, see the search options page. This example uses ["visual"].

from twelvelabs import TwelveLabs

CLASSES = [
    {
        "name": "DanceTok",
        "prompts": [
            "Dance tutorial",
            "Dance group",
            "Dance competition"
        ]
    },
    {
        "name": "CookTok",
        "prompts": [
            "Cooking tutorial",
            "Cooking ustensils review"
        ]
    }
]
client = TwelveLabs(api_key="<YOUR_API_KEY>")
# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio}"
            )


res = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASSES,
)

print_page(res.data)

while True:
    try:
        next_page_data = next(res)
        print_page(next_page_data)
    except StopIteration:
        break

const res = await client.classify.index({
  indexId: INDEX_ID,
  options: ['visual', 'conversation'],
  classes: CLASSES,
  showDetailedScore: true
});

The output should look similar to the following one:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99
video_id=665dbe2cd22b3a3c97bf0bbb
  name=DanceTok score=95.75 duration_ratio=0.84

In the example output above, note that the data array has three elements, and each element contains information about a specific video.

Retrieve information about each matching video clip

The following example code sets the include_clips parameter to True to specify that the platform should retrieve detailed information about each matching video clip:

# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} clips={cl.clips.model_dump_json(indent=2)}"
            )
            
res = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASSES,
    include_clips=True
)

const res = await client.classify.index({
  indexId: INDEX_ID,
  options: ['visual'],
  classes: CLASSES,
  includeClips: true
});

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio} clips=${JSON.stringify(cl.clips, null, 2)}`
      );
    });
  });
}

The following example output has been truncated for brevity:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99 clips=[
  {
    "start": 148.84999999999997,
    "end": 306.84999999999997,
    "score": 85.95,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf08d22b3a3c97bf0bbe/149.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=c8d382e4249b31037e4b653926b988112edb15ddf90dd7dfc305ec9e4dd61f45"
  },
  {
    "start": 476.84999999999997,
    "end": 494.84999999999997,
    "score": 85.91,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf08d22b3a3c97bf0bbe/477.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=9ae347c85f9d3b31cb2afd81fddcdffdd82e45e6f8dac0746a1a6220c84e0605"
  },
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99 clips=[
  {
    "start": 771.4333333332886,
    "end": 873.5666666665386,
    "score": 85.54,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf30d22b3a3c97bf0bbf/772.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=df7cff3781f5f83f585c9c32f392bba0fa71e01e4bc2d8339a84e5699e574225"
  },

In this example, note that, for each video, the platform returns an array named clips. Each element of these arrays contains detailed information about a single video clip.

Retrieve a detailed score for each class

The following example code sets the show_detailed_score parameter to True to specify that the platform must retrieve the maximum, average, and normalized scores for each class:

# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} detailed_scores={cl.detailed_scores.model_dump_json(indent=2)}"
            )
res = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASSES,
    show_detailed_score=True
)

const res = await client.classify.index({
  indexId: INDEX_ID,
  options: ['visual'],
  classes: CLASSES,
  showDetailedScore: true
});

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio} detailedScores=${JSON.stringify(cl.detailedScores, null, 2)}`
      );
    });
  });
}

The output should look similar to the following one:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99 detailed_scores={
  "max_score": 85.95,
  "avg_score": 83.94,
  "normalized_score": 100.0
}
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99 detailed_scores={
  "max_score": 85.54,
  "avg_score": 82.9,
  "normalized_score": 100.0
}
video_id=665dbd90d22b3a3c97bf0bb9
  name=DanceTok score=95.94 duration_ratio=0.99 detailed_scores={
  "max_score": 85.55,
  "avg_score": 80.99,
  "normalized_score": 100.0
}

Filtering and pagination

Understanding and utilizing filtering and pagination enhances your ability to find and organize information. See the sections below for more details: