Guides
SDKsSample appsIntegrationsDiscordPlayground

Classify a set of videos

Suggest Edits

This guide shows how you can classify a set of videos.

Prerequisites

The examples in this guide assume the following:

  • You’re familiar with the concepts that are described on the Platform overview page.
  • You've already created an index, and the Marengo video understanding engine is enabled for this index.
  • You've uploaded one or more videos, and the platform has finished indexing them. The unique identifiers of your videos are stored in a set of variables named VIDEO_ID_1 to VIDEO_ID_X. For details, see the Upload videos page.

Examples

This section shows several examples of classifying videos and specifying various parameters. For a description of each field in the request and response, see the API Reference > Classify a set of videos page.

Classify based on visual cues

To classify a set of videos based on visual cues, invoke the videos method of the classify object with the following parameters:

  • video_ids: An array of strings representing the unique identifiers of the videos you wish to classify.
  • classes: An array of objects containing the names and definitions of the entities or actions that the platform must identify.
  • options: An array of strings specifying the sources of information the platform uses when categorizing a video. For details, see the search options page. This example uses ["visual"].
from twelvelabs import TwelveLabs

VIDEO_IDS= [VIDEO_ID_1, VIDEO_ID_2, VIDEO_ID_2, VIDEO_ID_3]

CLASSES = [
    {
        "name": "DanceTok",
        "prompts": [
            "Dance tutorial",
            "Dance group",
            "Dance competition"
        ]
    },
    {
        "name": "CookTok",
        "prompts": [
            "Cooking tutorial",
            "Cooking ustensils review"
        ]
    }
]

client = TwelveLabs("<YOUR_API_KEY">)
# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio}"
            )


res = client.classify.videos(
    video_ids=VIDEO_IDS,
    options=["visual"],
    classes=CLASSES,
)

print_page(res.data)

while True:
    try:
        next_page_data = next(res)
        print_page(next_page_data)
    except StopIteration:
        break

import { TwelveLabs } from 'twelvelabs-js';

const VIDEO_IDS= [VIDEO_ID_1, VIDEO_ID_2, VIDEO_ID_3]

const CLASSES = [
  {
    name: 'DanceTok',
    prompts: ['Dance tutorial', 'Dance group', 'Dance competition'],
  },
  {
    name: 'CookTok',
    prompts: ['Cooking tutorial', 'Cooking ustensils review'],
  },
];

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>' });

const res = await client.classify.videos({
  videoIds: VIDEO_IDS,
  options: ['visual'],
  classes: CLASSES,
});

printPage(res);

let nextIndexPage = await res.next();
while (nextIndexPage !== null) {
  printPage(nextIndexPage);
  nextIndexPage = await res.next();
}

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio}`
      );
    });
  });
}

The output should look similar to the following one:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99
video_id=665dbe2cd22b3a3c97bf0bbb
  name=DanceTok score=95.75 duration_ratio=0.84

In this example, note that the data array has three elements, and each element contains information about a specific video.

Retrieve information about each matching video clip

The following example code sets the include_clips parameter to True to specify that the platform should retrieve detailed information about each matching video clip:

# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} clips={cl.clips.model_dump_json(indent=2)}"
            )
res = client.classify.videos(
    video_ids=VIDEO_IDS,
    options=["visual"],
    classes=CLASSES,
    include_clips=True
)

const res = await client.classify.videos({
  videoIds: VIDEO_IDS,
  options: ['visual'],
  classes: CLASSES,
  includeClips: true
});

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio} clips=${JSON.stringify(cl.clips, null, 2)}`
      );
    });
  });
}

The following example output has been truncated for brevity:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99 clips=[
  {
    "start": 148.84999999999997,
    "end": 306.84999999999997,
    "score": 85.95,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf08d22b3a3c97bf0bbe/149.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=c8d382e4249b31037e4b653926b988112edb15ddf90dd7dfc305ec9e4dd61f45"
  },
  {
    "start": 476.84999999999997,
    "end": 494.84999999999997,
    "score": 85.91,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf08d22b3a3c97bf0bbe/477.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=9ae347c85f9d3b31cb2afd81fddcdffdd82e45e6f8dac0746a1a6220c84e0605"
  },
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99 clips=[
  {
    "start": 771.4333333332886,
    "end": 873.5666666665386,
    "score": 85.54,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf30d22b3a3c97bf0bbf/772.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=df7cff3781f5f83f585c9c32f392bba0fa71e01e4bc2d8339a84e5699e574225"
  },

In this example, note that, for each video, the platform returns an array named clips. Each element of these arrays contains detailed information about a single video clip.

Retrieve a detailed score for each class

The following example code sets the show_detailed_score parameter to True to specify that the platform must retrieve the maximum, average, and normalized scores for each class:

# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} detailed_scores={cl.detailed_scores.model_dump_json(indent=2)}"
            )

res = client.classify.videos(
    video_ids=VIDEO_IDS,
    options=["visual"],
    classes=CLASSES,
    show_detailed_score=True
)

const res = await client.classify.videos({
  videoIds: VIDEO_IDS,
  options: ['visual'],
  classes: CLASSES,
  showDetailedScore: true
});

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio} detailedScores=${JSON.stringify(cl.detailedScores, null, 2)}`
      );
    });
  });
}

The output should look similar to the following one:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99 detailed_scores={
  "max_score": 85.95,
  "avg_score": 83.94,
  "normalized_score": 100.0
}
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99 detailed_scores={
  "max_score": 85.54,
  "avg_score": 82.9,
  "normalized_score": 100.0
}
video_id=665dbd90d22b3a3c97bf0bb9
  name=DanceTok score=95.94 duration_ratio=0.99 detailed_scores={
  "max_score": 85.55,
  "avg_score": 80.99,
  "normalized_score": 100.0
}

Filtering and pagination

Understanding and utilizing filtering and pagination enhances your ability to find and organize information. See the sections below for more details:

Updated 11 days ago