PricingDiscordPlayground

Classify a set of videos

Suggest Edits

This guide shows how you can classify a set of videos.

You can interact with the platform using one of the available SDKs or an HTTP client like requests or axios. This guide demonstrates how to use the SDKs, the recommended approach for most scenarios. If you need to make direct HTTP requests, refer to the API Reference > Classify a set of videos page for details.

Prerequisites

The examples in this guide assume the following:

  • You’re familiar with the concepts that are described on the Platform overview page.
  • You've already created an index, and the Marengo video understanding engine is enabled for this index.
  • You've uploaded one or more videos, and the platform has finished indexing them. The unique identifiers of your videos are stored in a set of variables named VIDEO_ID_1 to VIDEO_ID_X. For details, see the Upload videos page.

Examples

This section shows several examples of classifying videos and specifying various parameters. For a description of each field in the request and response, see the API Reference > Classify a set of videos page.

Classify based on visual cues

To classify a set of videos, invoke the videos method of the classify object with the following parameters:

  • video_ids: An array of strings representing the unique identifiers of the videos you wish to classify.
  • classes: An array of objects containing the names and definitions of the entities or actions that the platform must identify.
  • options: An array of strings specifying the sources of information the platform uses when categorizing a video. For details, see the search options page. This example uses ["visual"].
from twelvelabs import TwelveLabs

VIDEO_IDS= [VIDEO_ID_1, VIDEO_ID_2, VIDEO_ID_2, VIDEO_ID_3]

CLASSES = [
    {
        "name": "DanceTok",
        "prompts": [
            "Dance tutorial",
            "Dance group",
            "Dance competition"
        ]
    },
    {
        "name": "CookTok",
        "prompts": [
            "Cooking tutorial",
            "Cooking ustensils review"
        ]
    }
]

client = TwelveLabs("<YOUR_API_KEY">)
# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio}"
            )


res = client.classify.videos(
    video_ids=VIDEO_IDS,
    options=["visual],
    classes=CLASSES,
)

print_page(res.data)

while True:
    try:
        next_page_data = next(res)
        print_page(next_page_data)
    except StopIteration:
        break

import { TwelveLabs } from 'twelvelabs-js';

const VIDEO_IDS= [VIDEO_ID_1, VIDEO_ID_2, VIDEO_ID_3]

const CLASSES = [
  {
    name: 'DanceTok',
    prompts: ['Dance tutorial', 'Dance group', 'Dance competition'],
  },
  {
    name: 'CookTok',
    prompts: ['Cooking tutorial', 'Cooking ustensils review'],
  },
];

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>' });

const res = await client.classify.videos({
  videoIds: VIDEO_IDS,
  options: ['visual'],
  classes: CLASSES,
});

printPage(res);

let nextIndexPage = await res.next();
while (nextIndexPage !== null) {
  printPage(nextIndexPage);
  nextIndexPage = await res.next();
}

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio}`
      );
    });
  });
}

The output should look similar to the following one:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99
video_id=665dbe2cd22b3a3c97bf0bbb
  name=DanceTok score=95.75 duration_ratio=0.84

In the example output above, note that the data array has three elements, and each element contains information about a specific video.

Retrieve information about each matching video clip

The following example code sets the include_clips parameter to True to specify that the platform should retrieve detailed information about each matching video clip:

# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} clips={cl.clips.model_dump_json(indent=2)}"
            )
res = client.classify.videos(
    video_ids=VIDEO_IDS,
    options=["visual"],
    classes=CLASSES,
    include_clips=True
)

const res = await client.classify.videos({
  videoIds: VIDEO_IDS,
  options: ['visual'],
  classes: CLASSES,
  includeClips: true
});

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio} clips=${JSON.stringify(cl.clips, null, 2)}`
      );
    });
  });
}

The following example output has been truncated for brevity:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99 clips=[
  {
    "start": 148.84999999999997,
    "end": 306.84999999999997,
    "score": 85.95,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf08d22b3a3c97bf0bbe/149.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=c8d382e4249b31037e4b653926b988112edb15ddf90dd7dfc305ec9e4dd61f45"
  },
  {
    "start": 476.84999999999997,
    "end": 494.84999999999997,
    "score": 85.91,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf08d22b3a3c97bf0bbe/477.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=9ae347c85f9d3b31cb2afd81fddcdffdd82e45e6f8dac0746a1a6220c84e0605"
  },
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99 clips=[
  {
    "start": 771.4333333332886,
    "end": 873.5666666665386,
    "score": 85.54,
    "option": "visual",
    "prompt": "Dance competition",
    "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/665dbf30d22b3a3c97bf0bbf/772.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHW6XGLKFV%2F20240619%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240619T041830Z&X-Amz-Expires=604799&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=df7cff3781f5f83f585c9c32f392bba0fa71e01e4bc2d8339a84e5699e574225"
  },

In the example output above, note that, for each video, the platform returns an array named clips. Each element of these arrays contains detailed information about a single video clip.

Retrieve a detailed score for each class

The following example code sets the show_detailed_score parameter to True to specify that the platform must retrieve the maximum, average, and normalized scores for each class:

# Utility function to print a specific page
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} detailed_scores={cl.detailed_scores.model_dump_json(indent=2)}"
            )

res = client.classify.videos(
    video_ids=VIDEO_IDS,
    options=["visual"],
    classes=CLASSES,
    show_detailed_score=True
)

const res = await client.classify.videos({
  videoIds: VIDEO_IDS,
  options: ['visual'],
  classes: CLASSES,
  showDetailedScore: true
});

// Utility function to print a specific page
function printPage(result: any): void {
  result.data.forEach((data: any) => {
    console.log(`video_id=${data.videoId}`);
    data.classes.forEach((cl: any) => {
      console.log(
        `  name=${cl.name} score=${cl.score} durationRatio=${cl.durationRatio} detailedScores=${JSON.stringify(cl.detailedScores, null, 2)}`
      );
    });
  });
}

The output should look similar to the following one:

video_id=665dbf08d22b3a3c97bf0bbe
  name=DanceTok score=96.4 duration_ratio=0.99 detailed_scores={
  "max_score": 85.95,
  "avg_score": 83.94,
  "normalized_score": 100.0
}
video_id=665dbf30d22b3a3c97bf0bbf
  name=DanceTok score=96.03 duration_ratio=0.99 detailed_scores={
  "max_score": 85.54,
  "avg_score": 82.9,
  "normalized_score": 100.0
}
video_id=665dbd90d22b3a3c97bf0bb9
  name=DanceTok score=95.94 duration_ratio=0.99 detailed_scores={
  "max_score": 85.55,
  "avg_score": 80.99,
  "normalized_score": 100.0
}

Filtering and pagination

Understanding and utilizing filtering and pagination enhances your ability to find and organize information. See the sections below for more details:

Updated about 1 month ago