Open-ended text

Use the /generate endpoint for generating open-ended texts from videos that are more customizable and tailor-made than the results provided by the /gist and /summarize endpoints. This endpoint can generate diverse results based on your prompts, including, but not limited to, tables of content, action items, memos, reports, and comprehensive analyses.

Below are some examples of prompts tailored to generate specific content types:

Content typePrompt example
Table of contentsProvide a table of contents detailing the main sections of this video.
Action itemsIdentify and list all the action items assigned to each team member.
MemoGenerate a company-wide memo based on the announcements made in the video.
Police reportWrite a police report based on this video using the following example:

Date: 11/01/2020
Location: San Francisco Police Department
Witnesser’s full name: John Smith
Reporter: Barbara Lim
On 11/01/2020 around 5 PM, I saw a suspect walking in a retail store on Height Street…
Meeting minutesGenerate detailed meeting minutes from this video, including discussion points, decisions made, and follow-up actions assigned.
Video annotationsIdentify and list key visual elements, scene changes, and notable events in the video, briefly describing each.

For a description of each field in the request and response, see the API Reference > Generate an open-ended text representationpage.

📘

Note:

This endpoint is currently in early beta. Therefore, Twelve Labs recommends you validate the results before relying on them for critical applications. For feedback or suggestions, please use the messenger application at the bottom right corner of the Playground or email us at [email protected].

Prerequisites

The examples in this guide assume the following:

  • You’re familiar with the concepts that are described on the Platform overview page.
  • You’ve already created an index and the Pegasus video understanding engine is enabled for this index.
  • You've uploaded a video, and the platform has finished indexing it. The unique identifier of your video is stored in a variable named VIDEO_ID. For details about uploading videos, see the Upload videos page.

Examples

The following example generates brief summary with a specific format:

GENERATE_URL = f"{API_URL}/generate"

data = {
  "video_id": VIDEO_ID,
  "prompt": "I want to generate a description for my video with the following format: Title of the video, followed by a summary in 2-3 sentences, highlighting the main topic, key events, and concluding remarks."
}

response = requests.post(GENERATE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const GENERATE_URL = `${API_URL}/generate`

const data = {
    'video_id': VIDEO_ID,
    'prompt': 'I want to generate a description for my video with the following format: Title of the video, followed by a summary in 2-3 sentences, highlighting the main topic, key events, and concluding remarks.'
}

const config = {
    method: 'post',
    url: GENERATE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should be similar to the following one:

{
  "id": "f8db5a0b-414a-4aa0-bfc9-3574f3002956",
  "data": "Title: \"A Summer Day in Minnesota: College Graduation, Sun, Shopping, and Pennyboarding\"\n\nSummary: In this video, a woman shares her summer day in Minnesota after her college graduation. She vlogs about her temporary move back home, showing her childhood home and expressing her love for getting some sun. The video captures various activities, including applying sunscreen, discovering a foul smell in her car, a shopping haul from favorite stores, the preparation of a bread salad, and meeting up with a friend to go pennyboarding at a parking garage. It's a fun and eventful day filled with sunshine, shopping, and outdoor adventures."
}

The following example generates a police report based on the provided template:

GENERATE_URL = f"{API_URL}/generate"

data = {
  "video_id": VIDEO_ID,
  "prompt": "Write a police report based on this video with the following example:\nDate: \n11/01/2020\nLocation: San Francisco Police Department\nWitnesser’s full name: John Smith\nReporter: Barbara Lim\n\nOn 11/01/2020 around 5 PM, I saw a suspect walking in a retail store on Height Street…"
}

response = requests.post(GENERATE_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())
const GENERATE_URL = `${API_URL}/generate`

const data = {
    'video_id': VIDEO_ID,
    'prompt': 'Write a police report based on this video with the following example:\nDate: \n11/01/2020\nLocation: San Francisco Police Department\nWitnesser’s full name: John Smith\nReporter: Barbara Lim\n\nOn 11/01/2020 around 5 PM, I saw a suspect walking in a retail store on Height Street…'
}

const config = {
    method: 'post',
    url: GENERATE_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should be similar to the following one:

{
  "id": "4fe4ad58-6ef1-4386-85a2-0e3889871104",
  "data": "Date: 11/01/2020\nLocation: San Francisco Police Department\nWitness's full name: John Smith\nReporter: Barbara Lim\n\nOn 11/01/2020 around 5 PM, I, John Smith, witnessed a suspect walking in a retail store on Height Street. The suspect was observed stealing items from the store, including an item directly from the cash register. Two other individuals were also seen engaging in theft within the store.\n\nThe video evidence obtained from the store's surveillance cameras clearly captures the suspect's actions. The suspect was seen walking through the store and discreetly taking items without being noticed by anyone. Additionally, the video shows two other individuals stealing multiple items from the store before leaving.\n\nOne particular moment in the video shows a woman entering the camera's view, picking up a bottle of alcohol from the shelf, and putting it inside her bag. This incident adds to the evidence of theft within the store.\n\nBased on the video footage and witness testimony, it is evident that multiple instances of theft occurred within the retail store on Height Street. The stolen items include those taken directly from the cash register, as well as various other items throughout the store.\n\nWe request further investigation into this matter to identify and apprehend the suspects involved in these thefts. The video evidence should be analyzed thoroughly to assist in the identification and prosecution of the individuals responsible.\n\nWitnesser: John Smith\nReporter: Barbara Lim"
}