Request structured JSON responses when you analyze video content. This feature returns predictable, machine-readable data that integrates directly into your applications.

Key features:

• Control response structure: Define exactly how you want your video analysis results formatted using JSON schemas.
• Process responses programmatically: Eliminate manual parsing by receiving data in predictable, machine-readable formats.
• Validate data automatically: Ensure response fields meet your requirements with built-in type and format validation.
• Stream structured content: Retrieve real-time JSON responses as analysis progresses without waiting for completion.
• Integrate seamlessly: Connect video analysis directly into your existing data pipelines and applications.

Use cases:

• Directly populate data into databases: Store video analysis results in structured tables without manual formatting.
• Create automated reports: Generate consistent video summaries and insights for business intelligence dashboards.
• Build content recommendation systems: Extract keywords and topics to power personalized content suggestions.
• Populate content management systems: Auto-generate video titles, descriptions, and tags in your CMS workflow.
• Facilitate search and filtering: Organize video metadata to make content discoverable across your platform

Control responses

Use prompts and schemas together to control your responses:

• The prompt specifies what the platform must extract from the video.
• The schema defines the structure, field names, and data types for the response.

The schema takes precedence over the prompt. If your prompt requests a summary but your schema defines chapters, the platform returns chapters. Always ensure your prompt and schema align to request the same type of information.

Choose your schema

A schema defines:

• Field names: The properties that appear in the JSON response.
• Data types: The type of each field (example: string, number, or object).
• Required fields: Fields that must be present in every response.
• Nested structures: Complex data using object definitions and references.

For complete specifications, see the JSON schema requirements section.

Prerequisites

• You already know how to analyze videos and generate text based on their content. For instructions, see the Analyze videos guide.

Examples

The examples in this section show different types of structured responses you can request.

1
import json
2
from twelvelabs import TwelveLabs
3
from twelvelabs.types import AnalyzePromptV2, SyncResponseFormat, VideoContext_AssetId
4
5
text = client.analyze(
6
    model_name="pegasus1.5",
7
    video=VideoContext_AssetId(asset_id="<YOUR_ASSET_ID>"),
8
    prompt_v_2=AnalyzePromptV2(
9
        input_text="Provide a comprehensive summary of this video.",
10
    ),
11
    response_format=SyncResponseFormat(
12
        type="json_schema",
13
        json_schema={
14
            "type": "object",
15
            "properties": {
16
                "summary": {"type": "string"}
17
            },
18
            "required": ["summary"],
19
        },
20
    ),
21
)
22
23
data = json.loads(text.data) if text.data else {}
24
print(f"\n{data.get('summary', 'N/A')}\n")
25
print(f"\nFinish Reason: {text.finish_reason}")

Stream structured responses

You can receive structured JSON responses in real-time as they are generated:

1
from twelvelabs import TwelveLabs
2
from twelvelabs.types import AnalyzePromptV2, SyncResponseFormat, VideoContext_AssetId, StreamAnalyzeResponse_StreamEnd
3
4
text_stream = client.analyze_stream(
5
    model_name="pegasus1.5",
6
    video=VideoContext_AssetId(asset_id="<YOUR_ASSET_ID>"),
7
    prompt_v_2=AnalyzePromptV2(
8
        input_text="<YOUR_PROMPT>",
9
    ),
10
    response_format=SyncResponseFormat(
11
        type="json_schema",
12
        json_schema={
13
            "type": "object",
14
            "properties": {
15
                "title": {"type": "string"},
16
                "summary": {"type": "string"},
17
                "keywords": {"type": "array", "items": {"type": "string"}},
18
            },
19
        },
20
    ),
21
)
22
for chunk in text_stream:
23
    if chunk.event_type == "text_generation":
24
        print(chunk.text, end="", flush=True)
25
    elif isinstance(chunk, StreamAnalyzeResponse_StreamEnd):
26
        print(f"\nFinish reason: {chunk.finish_reason}")
27
        if chunk.metadata and chunk.metadata.usage:
28
            print(f"Usage: {chunk.metadata.usage}")

Best practices

• Start with simple schemas: Begin with basic object structures and add complexity only when needed.
• Keep schema fields focused and related: Schemas with unrelated fields may require multiple API calls to generate complete results. Group related analysis tasks together or make separate calls for different types of analyses.
• Keep field names descriptive: Use clear, consistent naming conventions for schema properties to make your code more maintainable.
• Mark essential fields as required: Use the required property to specify fields that must be present. This ensures critical data is always included in responses.
• Handle truncated responses: For open-ended analysis, always check the finish_reason field in your code. When it equals length, inspect the error field. Increase the maximum response length if your response needs more room, or reduce your input if the context window was reached.
• Validate responses client-side: Parse and validate the JSON response in your application to catch any formatting issues before processing the data.
• Test schemas thoroughly: Validate your schema with sample data before deploying to production. Use different video types and content lengths to ensure reliability.

Troubleshooting

This section provides solutions to common issues you might encounter when working with structured JSON responses.

Prompt and response mismatch

Problem: The response contains information different from what you requested in the prompt (for example, you asked for a summary but received chapters).

Cause: The schema takes precedence over the prompt. The platform generates content that matches the schema structure, regardless of what the prompt requests.

Solution: Ensure your prompt and schema align to request the same type of information.

Incomplete or invalid JSON

Problem: The response contains truncated JSON or fails to parse.

Cause: The response reached the maximum response length or the context window before completing the JSON structure.

Solution: Check the finish_reason field. When it equals length, the platform returns a warning in the error field, and the truncated response is in data (sync) or result.data (async). Increase the maximum response length if your response needs more room, or reduce your input if the context window was reached.

Invalid schema error

Problem: You receive a BadRequestError with code response_format_invalid when making a request.

Cause: The schema contains unsupported constraints (such as minLength or maxLength) or invalid references (such as $ref pointing to non-existent definitions).

Solution: Review the json_schema field in the API Reference section for a list of the supported constraints. Ensure all $ref values point to definitions in $defs. Remove unsupported validation keywords and use only the supported data types and constraints.

JSON schema requirements

Your schema must adhere to the JSON Schema Draft 2020-12 specification and must meet the requirements below:

• Supported data types: array, boolean, integer, null, number, object, and string.
• Schema constraints: Use validation keywords like pattern for strings, minimum and maximum for integers, required for objects, and minItems for arrays (accepts only 0 or 1).
• Schema composition: You can only use anyOf for combining schemas.
• References: Define subschemas in $defs and reference them with valid $ref URIs pointing to internal subschemas.

For complete schema specifications, see the json_schema field in the API Reference section.