Summaries, chapters, and highlights

Use the /summarize endpoint if you want to utilize pre-defined formats for general summarization tasks and, optionally, provide a prompt to customize the output. This endpoint generates the following:

  • Summaries: The platform returns a brief that encapsulates the key points of a video, presenting the most important information clearly and concisely. Depending on your prompt, a summary can take various forms, such as a single paragraph, a series of paragraphs, an email, or a structured list of bullet points. For example, a summary highlighting Black Friday events might include a description of a crowded mall, key commentary by a news reporter on consumer behavior, and individual perspectives on societal values associated with consumption.
  • Chapters: The platform returns a chronological list of all the chapters in a video, providing a granular breakdown of its content. For each chapter, the platform returns its starting and end times, measured in seconds from the beginning of the video, a descriptive headline that offers a brief of the events or activities within that chapter, and an accompanying summary that elaborates on the headline. For example, the first chapter of a stand-up comedy might describe the comedian's entrance and the first joke. The accompanying summary could delve into the content, detailing the comedian's humorous take on a specific subject, such as the cultural nuances of Tai Chi exercises.
  • Highlights: The platform returns a chronologically ordered list of the most important events within a video. Unlike chapters, highlights only capture the key moments, providing a snapshot of the video's main topics. For each highlight, the platform returns its starting and end times, measured in seconds from the beginning of the video, a title, and a brief that captures its essence. For example, a highlight might capture a significant event like a bank heist in a video with multiple scenes.

Below are some ways in which you can guide the platform to generate content tailored to your specific needs:

Content typePrompt example
Specify the target audienceGenerate a summary suitable for a high school audience studying environmental science.
Adjust the toneGenerate a light-hearted and humorous chapter breakdown of this documentary.
Indicate length constraintsProvide a summary fit for a Twitter post under 280 characters.
Customize text formatGenerate a summary in no more than 5 bullet points.
Specify the purposeSummarize this video from a marketer's perspective, focusing on brand mentions and product placements.

For a description of each field in the request and response, see the API Reference > Generate a summary section.

📘

Note:

This endpoint is currently in early beta. Therefore, Twelve Labs recommends you validate the results before relying on them for critical applications. For feedback or suggestions, please use the messenger application at the bottom right corner of the Playground or email us at [email protected].

Prerequisites

The examples in this guide assume the following:

  • You’re familiar with the concepts that are described on the Platform overview page.
  • You’ve already created an index and the Pegasus video understanding engine is enabled for this index.
  • You've uploaded a video, and the platform has finished indexing it. The unique identifier of your video is stored in a variable named VIDEO_ID. For details about uploading videos, see the Upload videos page.

Generate a summary

The following example code generates a summary for the specified video:

SUMMARIZE_URL = f"{API_URL}/summarize"

data = {
  "video_id": VIDEO_ID,
  "type": "summary"
}

response = requests.post(SUMMARIZE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const SUMMARIZE_URL = `${API_URL}/summarize`

const data = {
    'video_id': VIDEO_ID,
    'type': 'summary'
}

const config = {
    method: 'post',
    url: SUMMARIZE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should look similar to the following one:

Status code: 200
{
  "id": "81ea5d6d-33f1-4ef4-8105-bb7be386f7e5",
  "summary": "The video begins with a woman discussing her recent lifestyle changes, including her graduation from college and moving back home. She decides to vlog a summer day in her house in Minnesota, showing viewers where she grew up. She talks about her love for sunbathing, applying her tanning oil, and enjoying the outdoor environment. Afterward, she shares her outfit including her favorite tank from Hollister and fold-over pants.\n She journeys to the Mall of America, making a pit stop at Starbucks. At the mall, she shops at her favorite stores, including Hollister, Aritzia, and Victoria's Secret. She purchases various items including a dainty blue floral top, gray yoga pants, and new bras and shares her shopping haul with the viewers. The woman enjoys a sandwich, revealing her love for the food.\n There's a quick transition to her preparing dinner at home. She decides to whip up a tomato pasta dish called panzanella, using her mother's recipe. After letting it marinate for a bit, she samples the mix, recalling its deliciousness from her memories. \n In the evening, she meets her best friend Rachel for an outdoor adventure. They go pennyboarding, a favored activity of theirs, which they usually do in certain mall parking lots. She continues her activities with her friend, expressing her frustrations over her troublesome skateboard. The video concludes with the woman sharing her day, even as transcription becomes unavailable."
}

Optionally, you can use the prompt field to provide context for the summarization task. The following example specifies the purpose and the desired length:

SUMMARIZE_URL = f"{API_URL}/summarize"

data = {
  "video_id": VIDEO_ID,
  "type": "summary",
  "prompt": "Generate a summary of this video for a social media post, up to two sentences."
}

response = requests.post(SUMMARIZE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const SUMMARIZE_URL = `${API_URL}/summarize`

const data = {
    'video_id': VIDEO_ID,
    'type': 'summary',
    'prompt': 'Generate a summary of this video for a social media post, up to two sentences.'
}

const config = {
    method: 'post',
    url: SUMMARIZE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should look similar to the following one:

Status code: 200
{
  "id": "900b5e60-c99d-412e-938a-14e6dfb2bf77",
  "summary": "Two women share a day in their lives, discussing topics such as graduation, moving home, shopping hauls, cooking, and outdoor adventures. They showcase their favorite clothing items, prepare food, and go pennyboarding together."
}

Generate chapters

The following example generates a list of chapters for the specified video:

SUMMARIZE_URL = f"{API_URL}/summarize"

data = {
  "video_id": VIDEO_ID,
  "type": "chapter"
}

response = requests.post(SUMMARIZE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const SUMMARIZE_URL = `${API_URL}/summarize`

const data = {
    'video_id': VIDEO_ID,
    'type': 'chapter'
}

const config = {
    method: 'post',
    url: SUMMARIZE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should look similar to the following one:

{
  "id": "c4c73461-6e2e-4092-846b-60383cb0cd8c",
  "chapters": [
    {
      "chapter_number": 0,
      "start": 0,
      "end": 90,
      "chapter_title": "College graduation and temporary move back home",
      "chapter_summary": "A woman discusses her recent college graduation and temporary move back home."
    },
    {
      "chapter_number": 1,
      "start": 90,
      "end": 210,
      "chapter_title": "Enjoying a summer day in Minnesota",
      "chapter_summary": "The woman vlogs about her summer day, sharing her experience while standing in her childhood home in Minnesota. She talks about her love for getting some sun and shows the camera around her house."
    },
    {
      "chapter_number": 2,
      "start": 210,
      "end": 300,
      "chapter_title": "Shopping at the Mall of America",
      "chapter_summary": "The woman explores the Mall of America, mentioning her favorite stores and sharing her shopping haul."
    },
    {
      "chapter_number": 3,
      "start": 300,
      "end": 360,
      "chapter_title": "Sharing favorite clothing items",
      "chapter_summary": "The woman showcases her favorite clothing items, discussing her new yoga pants and tops from Aritzia and Victoria's Secret."
    },
    {
      "chapter_number": 4,
      "start": 360,
      "end": 480,
      "chapter_title": "Food preparation and outdoor activities",
      "chapter_summary": "The woman demonstrates how to make a bread salad and later meets up with her friend Rachel to go pennyboarding at a parking garage."
    }
  ]
}

The following example uses the prompt parameter to indicate that the tone of voice should be casual and conversational, matching the vlogging style of the video:

SUMMARIZE_URL = f"{API_URL}/summarize"

data = {
  "video_id": VIDEO_ID,
  "type": "chapter",
  "prompt": "Generate chapters using casual and conversational language to match the vlogging style of the video."
}

response = requests.post(SUMMARIZE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const SUMMARIZE_URL = `${API_URL}/summarize`

const data = {
    'video_id': VIDEO_ID,
    'type': 'chapter',
    'prompt': 'Generate chapters using casual and conversational language to match the vlogging style of the video.'
}

const config = {
    method: 'post',
    url: SUMMARIZE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should look similar to the following one:

{
  "id": "c4c73461-6e2e-4092-846b-60383cb0cd8c",
  "chapters": [
    {
      "chapter_number": 0,
      "start": 0,
      "end": 90,
      "chapter_title": "College graduation and temporary move back home",
      "chapter_summary": "A woman discusses her recent college graduation and temporary move back home."
    },
    {
      "chapter_number": 1,
      "start": 90,
      "end": 210,
      "chapter_title": "Enjoying a summer day in Minnesota",
      "chapter_summary": "The woman vlogs about her summer day, sharing her experience while standing in her childhood home in Minnesota. She talks about her love for getting some sun and shows the camera around her house."
    },
    {
      "chapter_number": 2,
      "start": 210,
      "end": 300,
      "chapter_title": "Shopping at the Mall of America",
      "chapter_summary": "The woman explores the Mall of America, mentioning her favorite stores and sharing her shopping haul."
    },
    {
      "chapter_number": 3,
      "start": 300,
      "end": 360,
      "chapter_title": "Sharing favorite clothing items",
      "chapter_summary": "The woman showcases her favorite clothing items, discussing her new yoga pants and tops from Aritzia and Victoria's Secret."
    },
    {
      "chapter_number": 4,
      "start": 360,
      "end": 480,
      "chapter_title": "Food preparation and outdoor activities",
      "chapter_summary": "The woman demonstrates how to make a bread salad and later meets up with her friend Rachel to go pennyboarding at a parking garage."
    }
  ]
}

Generate highlights

The following example generates a list of the important events or activities within a video:

SUMMARIZE_URL = f"{API_URL}/summarize"

data = {
  "video_id": VIDEO_ID,
  "type": "highlight"
}

response = requests.post(SUMMARIZE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const SUMMARIZE_URL = `${API_URL}/summarize`

const data = {
    'video_id': VIDEO_ID,
    'type': 'highlight',
}

const config = {
    method: 'post',
    url: SUMMARIZE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should be similar to the following one:

{
  "id": "e00dca53-4c9a-44b2-8a99-8427134cfb99",
  "highlights": [
    {
      "start": 30,
      "end": 60,
      "highlight": "A woman vlogs about her summer day, sharing her experience while standing in her childhood home in Minnesota.",
      "highlight_summary": "The woman talks about her love for getting some sun, despite not tanning well, and shows the camera around her house, mentioning her fake tanner."
    },
    {
      "start": 150,
      "end": 180,
      "highlight": "The woman talks about the heat and her confusion regarding sweat and tanning oil. She also mentions her outfit, consisting of a tank top and fold-over pants.",
      "highlight_summary": "The woman talks about the heat and her confusion regarding sweat and tanning oil. She also mentions her outfit, consisting of a tank top and fold-over pants."
    },
    {
      "start": 420,
      "end": 450,
      "highlight": "A woman is making a video of her food preparation and tasting.",
      "highlight_summary": "A woman demonstrates how to make a delicious bread salad with tomatoes and cheese while explaining the process."
    }
  ]
}

The following example uses the prompt parameter to generate highlights for the same video, showcasing the most entertaining parts:

SUMMARIZE_URL = f"{API_URL}/summarize"

data = {
  "video_id": VIDEO_ID,
  "type": "highlight",
  "prompt": "Generate highlights that showcase the most entertaining parts of the video."
}

response = requests.post(SUMMARIZE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const SUMMARIZE_URL = `${API_URL}/summarize`

const data = {
    'video_id': VIDEO_ID,
    'type': 'highlight',
  	'prompt': 'Generate highlights that showcase the most entertaining parts of the video.'
}

const config = {
    method: 'post',
    url: SUMMARIZE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should look similar to the following one:

{
  "id": "32f9e4f6-056e-496a-b3a3-b29c56882c8a",
  "highlights": [
    {
      "start": 120,
      "end": 150,
      "highlight": "The woman discovers a foul smell in her car while vlogging about her day.",
      "highlight_summary": "The woman in the car talks about her day and then discovers a foul smell in the car."
    },
    {
      "start": 210,
      "end": 240,
      "highlight": "The woman shares her experience of shopping in a mall and discusses a surprising incident that happened to her.",
      "highlight_summary": "A woman is sharing her experience of shopping in a mall and discussing a surprising incident that happened to her."
    }
  ]
}