Pegasus 1.5 introduces video segmentation. structured prompts with reference images, video clipping, and per-definition time ranges for segmentation. It supports both general analysis (prompt-based text generation) and video segmentation. Pegasus 1.2 remains available for general analysis only.

Warning

Upgrading the SDK introduces one breaking change: the ResponseFormat type has been split into SyncResponseFormat and AsyncResponseFormat. If you use structured responses, update your imports before upgrading. See Update the ResponseFormat type for details.

Except for the breaking change mentioned above, your existing code continues to work after upgrading. TwelveLabs recommends migrating to Pegasus 1.5 to access the new features.

Migration steps

Update your SDK

Install the latest version of the Python or Node.js SDK.

$ pip install --upgrade twelvelabs

Update the ResponseFormat type

If you use structured responses, the ResponseFormat type has been split into SyncResponseFormat (for sync analysis) and AsyncResponseFormat (for async analysis).

Before:

1 from twelvelabs.types import ResponseFormat
2 
3 result = client.analyze(
4     video=VideoContext_AssetId(asset_id="<YOUR_ASSET_ID>"),
5     prompt="Summarize this video.",
6     response_format=ResponseFormat(
7         type="json_schema",
8         json_schema={"type": "object", "properties": {"summary": {"type": "string"}}},
9     ),
10 )

After (sync):

1 from twelvelabs.types import SyncResponseFormat, AnalyzePromptV2
2 
3 result = client.analyze(
4     model_name="pegasus1.5",
5     video=VideoContext_AssetId(asset_id="<YOUR_ASSET_ID>"),
6     prompt_v_2=AnalyzePromptV2(input_text="Summarize this video."),
7     response_format=SyncResponseFormat(
8         type="json_schema",
9         json_schema={"type": "object", "properties": {"summary": {"type": "string"}}},
10     ),
11 )

Update your analyze calls

Make the following changes to your sync or async analyze calls:

Set model_name to "pegasus1.5". If you omit this parameter, the platform uses Pegasus 1.2 by default.
Replace the prompt string parameter with prompt_v2, a structured object. Set input_text to your prompt text. To include reference images, add media_sources.

The parameter names and usage are the same for both sync and async methods.

Before:

1 result = client.analyze(
2     video=VideoContext_AssetId(asset_id="<YOUR_ASSET_ID>"),
3     prompt="Summarize this video.",
4 )

After:

1 from twelvelabs.types import AnalyzePromptV2
2 
3 result = client.analyze(
4     model_name="pegasus1.5",
5     video=VideoContext_AssetId(asset_id="<YOUR_ASSET_ID>"),
6     prompt_v_2=AnalyzePromptV2(
7         input_text="Summarize this video.",
8     ),
9 )

Note

Pegasus 1.5 supports up to 65,536 tokens (compared to 4,096 for Pegasus 1.2). If your application validates or caps max_tokens, update the boundary.

Explore new capabilities

Pegasus 1.5 introduces the following capabilities that are not available in Pegasus 1.2.

Video clipping

Analyze a specific portion of a video using the start_time and end_time parameters. The clip must be at least 4 seconds long. The following example analyzes a 60-second clip starting at the 30-second mark:

1 result = client.analyze(
2     model_name="pegasus1.5",
3     video=VideoContext_Url(url="<YOUR_VIDEO_URL>"),
4     prompt_v_2=AnalyzePromptV2(
5         input_text="What happens in this clip?",
6     ),
7     start_time=30.0,
8     end_time=90.0,
9 )

Structured prompts with reference images

Reference up to 4 images in your prompt using the prompt_v_2 parameter. Use <@name> placeholders in the prompt text and declare the images in media_sources. The following example asks whether a reference photo matches a person in the video:

1 from twelvelabs.types import AnalyzePromptV2, SmeMediaSource, VideoContext_Url
2 
3 result = client.analyze(
4     model_name="pegasus1.5",
5     video=VideoContext_Url(url="<YOUR_VIDEO_URL>"),
6     prompt_v_2=AnalyzePromptV2(
7         input_text="Does the person in the video resemble <@reference_photo>?",
8         media_sources=[
9             SmeMediaSource(
10                 name="reference_photo",
11                 media_type="image",
12                 url="<YOUR_IMAGE_URL>",
13             ),
14         ],
15     ),
16 )

Video segmentation

Pegasus 1.5 transforms raw videos into structured, timestamped data. Define the types of segments you want to detect and the custom fields you want to extract. Video segmentation requires the asynchronous analysis endpoint. The following example detects scene changes and extracts a sentiment field for each segment:

1 from twelvelabs.types import (
2     VideoContext_Url,
3     AsyncResponseFormat,
4     SegmentDefinition,
5     SegmentField,
6     SegmentFieldItems,
7 )
8 
9 task = client.analyze_async.tasks.create(
10     model_name="pegasus1.5",
11     video=VideoContext_Url(url="<YOUR_VIDEO_URL>"),
12     analysis_mode="time_based_metadata",
13     response_format=AsyncResponseFormat(
14         type="segment_definitions",
15         segment_definitions=[
16             SegmentDefinition(
17                 id="scene",
18                 description="A distinct scene or setting change in the video",
19                 fields=[
20                     SegmentField(
21                         name="sentiment",
22                         type="string",
23                         description="The emotional tone of this segment",
24                         enum=["positive", "negative", "neutral"],
25                     ),
26                 ],
27             ),
28         ],
29     ),
30 )

Per-definition time ranges for segmentation

Restrict segment extraction to specific time windows within the video. Add the time_ranges parameter to individual segment definitions. The following example limits scene detection to two specific time windows (0–30s and 60–90s):

1 from twelvelabs.types import (
2     VideoContext_Url,
3     AsyncResponseFormat,
4     SegmentDefinition,
5     AnalyzeTimeRange,
6 )
7 
8 task = client.analyze_async.tasks.create(
9     model_name="pegasus1.5",
10     video=VideoContext_Url(url="<YOUR_VIDEO_URL>"),
11     analysis_mode="time_based_metadata",
12     response_format=AsyncResponseFormat(
13         type="segment_definitions",
14         segment_definitions=[
15             SegmentDefinition(
16                 id="scene",
17                 description="A distinct scene or setting change",
18                 time_ranges=[
19                     AnalyzeTimeRange(start_time=0.0, end_time=30.0),
20                     AnalyzeTimeRange(start_time=60.0, end_time=90.0),
21                 ],
22             ),
23         ],
24     ),
25 )

Additional resources

Analyze videos

Segment videos

Python SDK Reference

Node.js SDK Reference

API Reference