Pegasus is a generative model for video-to-text generation. Pegasus analyzes multiple modalities to generate contextually relevant text based on the content of your videos.

Available versions

Pegasus 1.6

Analyze videos or images to generate text, with egocentric video understanding, improved entity recognition, and in-segment events.

Pegasus 1.5

Analyze videos to generate summaries, detailed descriptions, and structured responses.

Examples

Segmenting egocentric video into atomic actions

This example request segments first-person factory footage into atomic actions and labels the hand used for each one. The output is truncated to the first three segments.

{
  "model_name": "pegasus1.6",
  "analysis_mode": "time_based_metadata",
  "video": {
    "type": "asset_id",
    "asset_id": "<YOUR_ASSET_ID>"
  },
  "temperature": 0,
  "min_segment_duration": 2,
  "max_tokens": 98304,
  "response_format": {
    "type": "segment_definitions",
    "segment_definitions": [
      {
        "id": "atomic_action",
        "description": "Cut this egocentric video into atomic actions. One segment is ONE uninterrupted motion by one hand, or by both hands when they act on the same object together. Cut a NEW segment at EVERY change in what a hand is doing — a new contact, a release, a re-grasp, a change of direction, a change of object, a change of which hand is acting, a correction, a nudge, a pause. Small and incidental motions count: a two-second re-grip is its own segment. Never summarise several motions into one segment, and never skip a motion because it is brief. A repeated activity is one segment per repetition, never one segment for the whole run. Cover the clip end to end — consecutive segments touch, each starting where the previous one ended, with no gaps and no overlaps — and hold the same level of detail in the last minute as in the first.",
        "fields": [
          {
            "name": "action",
            "type": "string",
            "description": "Exactly ONE action: the single motion this segment covers, as verb + object + hand. Never a list and never a sequence — if you are about to join two verbs with 'and' or a comma, that is two segments, not one. Name the motion, not its purpose."
          },
          {
            "name": "hands_used",
            "type": "string",
            "description": "Which hand performs THIS action. Use 'both' only when the two hands act on the same object at the same moment and neither could do it alone. If one hand acts while the other only holds, steadies, supports or waits, the value is the ACTING hand alone, not 'both'. Use 'none' when neither hand is manipulating anything. Asymmetric work is the norm, so do not default to 'both' because two hands are visible.",
            "enum": ["left", "right", "both", "none"]
          },
          {
            "name": "description",
            "type": "string",
            "description": "One sentence describing this single motion in full."
          }
        ]
      }
    ]
  }
}

Labeling task attempts in an egocentric cooking video

This example request segments a first-person cooking video into task attempts and states the goal of each attempt.

{
  "model_name": "pegasus1.6",
  "analysis_mode": "time_based_metadata",
  "video": {
    "type": "asset_id",
    "asset_id": "<YOUR_ASSET_ID>"
  },
  "temperature": 0,
  "min_segment_duration": 10,
  "max_tokens": 98304,
  "response_format": {
    "type": "segment_definitions",
    "segment_definitions": [
      {
        "id": "task",
        "description": "A coherent task attempt by the camera-wearer, from the first action that begins it to the point where its goal is achieved or abandoned. Do not split a task into its constituent motions — one segment per task attempt. Example boundaries: the whole of 'pack one shirt into a box', not the reach, the fold and the place separately.",
        "fields": [
          {
            "name": "goal",
            "type": "string",
            "description": "Imperative phrasing of the task goal, as a language command would be given to a robot, e.g. 'fold the shirt and place it in the box'. Keep it short — a command, not a narration. Under 60 characters, no 'the person…', no account of what happened."
          },
          {
            "name": "description",
            "type": "string",
            "description": "What the camera-wearer actually did, in one sentence: the objects involved and the order of the steps. Say what the goal does not already say — never restate the goal in the third person. Maximum 200 characters."
          }
        ]
      }
    ]
  }
}

Auditing privacy exposure in egocentric video

This example request flags faces, bystanders, documents, and screens in first-person factory footage for a privacy review. The output is truncated to one segment per definition.

{
  "model_name": "pegasus1.6",
  "analysis_mode": "time_based_metadata",
  "video": {
    "type": "asset_id",
    "asset_id": "<YOUR_ASSET_ID>"
  },
  "temperature": 0,
  "min_segment_duration": 2,
  "max_tokens": 98304,
  "response_format": {
    "type": "segment_definitions",
    "segment_definitions": [
      {
        "id": "face",
        "description": "A stretch of time in which a human face is visible in frame. Start a new segment each time a face enters frame and end it when no face remains. Do not create a segment when no face is visible.",
        "fields": [
          {
            "name": "identifiable",
            "type": "string",
            "description": "Whether the face is visible clearly enough and long enough that the person could be recognised from this footage. Use 'partially' when the face is turned, blurred by motion, or only briefly in frame.",
            "enum": ["yes", "partially", "no"]
          },
          {
            "name": "description",
            "type": "string",
            "description": "One sentence stating where in the frame the face appears and what makes it identifiable or not."
          }
        ]
      },
      {
        "id": "bystander",
        "description": "A stretch of time in which a person other than the camera-wearer is visible in frame, whether or not their face can be seen. Start a new segment each time such a person enters frame. Do not create a segment for the camera-wearer's own hands or body.",
        "fields": [
          {
            "name": "position",
            "type": "string",
            "description": "Where in the frame the person is, relative to the work the camera-wearer is doing.",
            "enum": ["foreground", "background", "edge-of-frame"]
          },
          {
            "name": "description",
            "type": "string",
            "description": "One sentence stating what is visible of this person and what they appear to be doing."
          }
        ]
      },
      {
        "id": "document",
        "description": "A stretch of time in which printed or handwritten text on paper, card, or a label is visible in frame. Do not create a segment for text on an electronic display.",
        "fields": [
          {
            "name": "legible",
            "type": "string",
            "description": "Whether the text could actually be read from this footage, as opposed to being recognisable as text.",
            "enum": ["yes", "partially", "no"]
          },
          {
            "name": "description",
            "type": "string",
            "description": "One sentence stating what the document appears to be and whether any of its text can be made out."
          }
        ]
      },
      {
        "id": "screen",
        "description": "A stretch of time in which an electronic display is visible in frame. Start a new segment each time a display enters frame.",
        "fields": [
          {
            "name": "legible",
            "type": "string",
            "description": "Whether what the display is showing could actually be read from this footage.",
            "enum": ["yes", "partially", "no"]
          },
          {
            "name": "description",
            "type": "string",
            "description": "One sentence stating what kind of display it is and what, if anything, can be made out on it."
          }
        ]
      }
    ]
  }
}

Summarizing educational videos

This example prompt summarizes an educational video without any customization.

Summarize this video

This example prompt generates a caption for a social media post.

Generate an attention-grabbing caption for a social media post. Keep it shorter than 200 characters.

This example prompt creates a table of contents detailing the main sections.

Provide a table of contents detailing the main sections of this video.

Company-wide memo

This example prompt generates a company-wide memo.

Generate a company-wide memo based on the content of this video.

Video annotations

This example prompt identifies and lists key visual elements, scene changes, and notable events, briefly describing each.

Identify and list key visual elements, scene changes, and notable events in the video, briefly describing each.

Video question answering

This example prompt identifies the key takeaways.

What are the key takeaways of this video?

Timestamp breakdown

This example prompt lists all timestamps in an advertisement where a close-up of the product appears.

Tell me all the timestamps in the advertisement where a closeup of the product appears.

Police report

This example prompt creates a police report using a specific template for a video showing a robbery.

Create a police report based on what happened in the video. Provide the exact time range where the suspect appears in the video

Using different languages

Spanish

This example prompt summarizes a video and indicates that the response should be in Spanish. Note that the prompt is in English, and the output is in Spanish.

Write a summary in Spanish.

French

This example prompt summarizes the three main takeaways of a video. Note that the prompt and the output are in French.

Résumez les trois principaux points à retenir de cette vidéo

Support

For support or feedback regarding Pegasus, contact support@twelvelabs.io.