Python SDK

Analyze videos

The TwelveLabs Python SDK provides methods to analyze videos and generate text from their content.

Related quickstart notebook

Methods

Open-ended analysis

Description: This method analyzes a video and generates text based on its content.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1def analyze(
2    self,
3    *,
4    video_id: str,
5    prompt: str,
6    temperature: typing.Optional[float] = OMIT,
7    response_format: typing.Optional[ResponseFormat] = OMIT,
8    max_tokens: typing.Optional[int] = OMIT,
9    request_options: typing.Optional[RequestOptions] = None,
10) -> NonStreamAnalyzeResponse:

Parameters

NameTypeRequiredDescription
video_idstrYesThe unique identifier of the video for which you wish to generate a text.
promptstrYesA prompt that guides the model on the desired format or content. Your prompts can be instructive or descriptive, or you can also phrase them as questions. The maximum length of a prompt is 2,000 tokens.
temperaturetyping.Optional[float]NoControls the randomness of the text output generated by the model. A higher value generates more creative text, while a lower value produces more deterministic text output. Default: 0.2, Min: 0, Max: 1
response_formattyping.Optional[ResponseFormat]NoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text.
max_tokenstyping.Optional[int]NoThe maximum number of tokens to generate. Min: 1. Max: 4096.
request_optionstyping.Optional[RequestOptions]NoRequest-specific configuration.

The ResponseFormat class contains the following properties:

NameTypeDescription
typeResponseFormatTypeSet this parameter to “json_schema” to receive structured JSON responses.
json_schematyping.Dict[str, typing.Optional[typing.Any]]Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value

Returns a NonStreamAnalyzeResponse object containing the generated text.

The NonStreamAnalyzeResponse class contains the following properties:

NameTypeDescription
idOptional[str]Unique identifier of the response.
dataOptional[str]The generated text based on the prompt you provided.
finish_reasontyping.Optional[FinishReason]The reason for the generation to finish. Values:
- null: The generation hasn’t finished yet.
- stop: The generation stopped because the model reached the maximum number of tokens. For JSON responses, this may result in truncated, invalid JSON that fails to parse.
usageOptional[TokenUsage]The number of tokens used in the generation.

The TokenUsage class contains the following properties:

NameTypeDescription
output_tokensOptional[int]The number of tokens in the generated text.

The maximum length of the response is 4,096 tokens.

API Reference

Open-ended analysis.

Analyze videos.

Open-ended analysis with streaming responses

Description: This method analyzes a video and generates open-ended text based on its content.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1def analyze_stream(
2    self,
3    *,
4    video_id: str,
5    prompt: str,
6    temperature: typing.Optional[float] = OMIT,
7    response_format: typing.Optional[ResponseFormat] = OMIT,
8    max_tokens: typing.Optional[int] = OMIT,
9    request_options: typing.Optional[RequestOptions] = None,
10) -> typing.Iterator[StreamAnalyzeResponse]

Parameters

NameTypeRequiredDescription
video_idstrYesThe unique identifier of the video for which you wish to generate a text.
promptstrYesA prompt that guides the model on the desired format or content. Your prompts can be instructive or descriptive, or you can also phrase them as questions. The maximum length of a prompt is 2,000 tokens.
temperatureOptional[float]NoControls the randomness of the text output generated by the model. A higher value generates more creative text, while a lower value produces more deterministic text output. Default: 0.2, Min: 0, Max: 1
response_formattyping.Optional[ResponseFormat]NoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text.
max_tokenstyping.Optional[int]NoThe maximum number of tokens to generate. Min: 1. Max: 4096.
request_optionsOptional[RequestOptions]NoRequest-specific configuration.

The ResponseFormat class contains the following properties:

NameTypeDescription
typeResponseFormatTypeSet this parameter to “json_schema” to receive structured JSON responses.
json_schematyping.Dict[str, typing.Optional[typing.Any]]Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value

Returns an iterator of StreamAnalyzeResponse objects. Each response can be a StreamAnalyzeResponse_StreamStart, StreamAnalyzeResponse_TextGeneration, or StreamAnalyzeResponse_StreamEnd.

The StreamAnalyzeResponse_StreamStart class contains the following properties:

NameTypeDescription
event_typeOptional[str]This field is always set to stream_start for this event.
metadataOptional[StreamStartResponseMetadata]An object containing metadata about the stream.

The StreamAnalyzeResponse_TextGeneration class contains the following properties:

NameTypeDescription
event_typeOptional[str]This field is always set to text_generation for this event.
textOptional[str]A fragment of the generated text. Note that text fragments may be split at arbitrary points, not necessarily at word or sentence boundaries.

The StreamAnalyzeResponse_StreamEnd class contains the following properties:

NameTypeDescription
event_typetyping.Literal["stream_end"]This field is always set to stream_end for this event.
metadatatyping.Optional[StreamEndResponseMetadata]An object containing metadata about the stream.
finish_reasontyping.Optional[FinishReason]The reason for the generation to finish. Values:
- null: The generation hasn’t finished yet.
- stop: The generation stopped because the model reached the maximum number of tokens. For JSON responses, this may result in truncated, invalid JSON that fails to parse.

The StreamStartResponseMetadata class contains the following properties:

NameTypeDescription
generation_idOptional[str]A unique identifier for the generation session.

The StreamEndResponseMetadata class contains the following properties:

NameTypeDescription
generation_idOptional[str]The same unique identifier provided in the stream_start event.
usageOptional[TokenUsage]The number of tokens used in the generation.

The TokenUsage class contains the following properties:

NameTypeDescription
output_tokensOptional[int]The number of tokens in the generated text.

The maximum length of the response is 4,096 tokens.

API Reference

Open-ended analysis.

Analyze videos.

Error codes

This section lists the most common error messages you may encounter while analyzing videos.

  • token_limit_exceeded
    • Your request could not be processed due to exceeding maximum token limit. Please try with another request or another video with shorter duration.
  • index_not_supported_for_generate
    • You can only summarize videos uploaded to an index with an engine from the Pegasus family enabled.