Classifying a single video

This section shows how to classify the content of a single video by calling the POST method of the /classify endpoint. For a description of each field in the request and response, see the API Reference > Classify a video page.

Prerequisites

The examples in this guide assume the following:

  • A valid Twelve Labs account. For details about creating an account and retrieving your API key, see the Authentication page.
  • You’re familiar with the concepts that are described on the Quickstart page.
  • You've uploaded a video, and the API service has finished indexing it. The unique identifier of your video is stored in a variable named VIDEO_ID. For details, see the Uploading videos page.

Examples

Note that, although the example code in this guide is written in Python and Node.js, the API is compatible with most programming languages, and you can also use Postman or other REST clients to send requests and view responses.

Classifying a video based on visual cues

The following example code defines a set of car shapes and uses the options parameter to specify that the API service should classify your video based on visual cues:

CLASSIFY_URL = f"{API_URL}/classify"
headers = {
    "x-api-key": API_KEY
}
data =  {
  "conversation_option": "semantic",
  "options": ["visual"],
  "video_id": VIDEO_ID,
  "threshold": "medium",
  "labels": [
      {
      "name": "Car shapes",
      "prompts": [
        "SUV",
        "Hatchback",
        "Crossover",
        "Sedan",
        "Coupe",
        "Minivan",
        "Station Wagon",
        "Pickup Truck",
        "Truck",
        "Convertible"
      ]
    }
  ]
}
response = requests.post(CLASSIFY_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())
const CLASSIFY_URL = `${API_URL}/classify`
const data =  {
  "conversation_option": "semantic",
  "options": ["visual"],
  "video_id": VIDEO_ID,
  "threshold": "medium",
  "labels": [
      {
      "name": "Car shapes",
      "prompts": [
        "SUV",
        "Hatchback",
        "Crossover",
        "Sedan",
        "Coupe",
        "Minivan",
        "Station Wagon",
        "Pickup Truck",
        "Truck",
        "Convertible"
      ]
    }
  ]
}
const resp = await axios.post(
  CLASSIFY_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))
}

The output should look similar to the following one

Status code: 200
{
    "video_id": "63a09373ce36463e0199c8de",
    "labels": [
        {
            "name": "Car shapes",
            "max_score": 87.15,
            "duration": 6069.999999999998,
            "confidence": "high"
        }
    ]
}

Note that, when classifying a video, the API service finds all video fragments that match the label you've specified in the request. For each video fragment found, the API service determines the level of confidence that the fragment matches the label. The max_score field is determined by comparing the confidence scores of each fragment and selecting the highest one.

Retrieving information about each matching video fragment

The following example code sets the include_clips parameter to true to specify that the API service should retrieve detailed information about each matching video fragment:

CLASSIFY_URL = f"{API_URL}/classify"
headers = {
    "x-api-key": API_KEY
}
data =  {
  "conversation_option": "semantic",
  "options": ["visual"],
  "video_id": VIDEO_ID,
  "include_clips": True,
  "labels": [
      {
      "name": "Car shapes",
      "prompts": [
        "SUV",
        "Hatchback",
        "Crossover",
        "Sedan",
        "Coupe",
        "Minivan",
        "Station Wagon",
        "Pickup Truck",
        "Truck",
        "Convertible"
      ]
    }
  ]
}

response = requests.post(CLASSIFY_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())
const CLASSIFY_URL = `${API_URL}/classify`
const data =  {
  "conversation_option": "semantic",
  "options": ["visual"],
  "video_id": VIDEO_ID,
  "include_clips": true,
  "labels": [
      {
      "name": "Car shapes",
      "prompts": [
        "SUV",
        "Hatchback",
        "Crossover",
        "Sedan",
        "Coupe",
        "Minivan",
        "Station Wagon",
        "Pickup Truck",
        "Truck",
        "Convertible"
      ]
    }
  ]
}
const resp = await axios.post(
  CLASSIFY_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following example output has been truncated for brevity:

{
  "video_id": "63a09373ce36463e0199c8de",
  "labels": [
    {
      "name": "Car Shapes",
      "max_score": 87.15,
      "duration": 288.8906249999999,
      "confidence": "high",
      "clips": [
        {
          "start": 9.25,
          "end": 12.78125,
          "score": 83.46,
          "confidence": "high",
          "option": "visual",
          "prompt": "Minivan",
          "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a09373ce36463e0199c8de/10.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221220%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221220T141434Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=43fd82d5e8e4b3f12fe3b7a6dc59e884c707fed1b8cee65f634c6acfb2bcd70f"
        },
        {
          "start": 65.5625,
          "end": 69.36458333333333,
          "score": 83.32,
          "confidence": "high",
          "option": "visual",
          "prompt": "Hatchback",
          "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a09373ce36463e0199c8de/66.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221220%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221220T141434Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=11270df806504589f72cc01d8e0a6fe1d2d244c8f41dea3bfab7a8e7ff4dac87"
        },
        {
          "start": 72.96875,
          "end": 75.96875,
          "score": 83.85,
          "confidence": "high",
          "option": "visual",
          "prompt": "SUV",
          "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a09373ce36463e0199c8de/73.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221220%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221220T141434Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=5c418a313fb81f73dd02297ee29f5c037bb0ad8dcae3b1e25b3315d75fc16fde"
        }
      ]
    }
  ]
}

In the example output above, note that each element in the clips array contains detailed information about a single matching video fragment.

Classifying a video based on multiple labels

The following example code classifies the content of a video based on two labels - Pets and Wild animals:

CLASSIFY_URL = f"{API_URL}/classify"
headers = {
    "x-api-key": API_KEY
}
data =  {
  "conversation_option": "semantic",
  "options": ["visual"],
  "threshold": "high",
  "video_id": "VIDEO_ID",
  "labels": [
      {
      "name": "Pets",
      "prompts": [
        "Cat",
        "Dog",
        "Guinea pig",
        "Parrot",
        "Goldfish"
      ]
    },
    {
      "name": "Wild Animals",
      "prompts": [
        "Crocodile",
        "Snake",
        "Hippo",
        "Lion",
        "Bear",
        "Wolf",
        "Lion",
        "Tiger"
      ]
    }
  ]
}
response = requests.post(CLASSIFY_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())
const CLASSIFY_URL = `${API_URL}/classify`
const  data =  {
  "conversation_option": "semantic",
  "options": ["visual"],
  "threshold": "high",
  "video_id": VIDEO_ID,
  "labels": [
      {
      "name": "Pets",
      "prompts": [
        "Cat",
        "Dog",
        "Guinea pig",
        "Parrot",
        "Goldfish"
      ]
    },
    {
      "name": "Wild Animals",
      "prompts": [
        "Crocodile",
        "Snake",
        "Hippo",
        "Lion",
        "Bear",
        "Wolf",
        "Lion",
        "Tiger"
      ]
    }
  ]
}
const resp = await axios.post(
  CLASSIFY_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The output should look similar to the following one:

Status code: 200
{
    "video_id": "63996260ce36463e0199c8c5",
    "labels": [
        {
            "name": "Pets",
            "max_score": 83.23,
            "duration": 4,
            "confidence": "high"
        },
        {
            "name": "Wild Animals",
            "max_score": 87.4,
            "duration": 531.2916666666666,
            "confidence": "high"
        }
    ]
}

In the example output above, note that the API service identified both the "Pets" and "Wild Animals" labels in the video. The maximum score is higher, and the duration is longer for the video fragments in which wild animals appear. This means that it is more likely this video contains wild animals rather than pets.

Filtering

This endpoint supports filtering. For details, see the Filtering > Content classification page.