Node.js SDK

Analyze videos

The TwelveLabs Node.js SDK provides methods to analyze videos to generate text from their content.

Open-ended analysis

Description: This method analyzes a video and generates text based on its content.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1analyze(request: TwelvelabsApi.AnalyzeRequest, requestOptions?: TwelvelabsApiClient.RequestOptions): core.HttpResponsePromise<TwelvelabsApi.NonStreamAnalyzeResponse>;

Parameters

NameTypeRequiredDescription
requestTwelvelabsApi.AnalyzeRequestYesThe request object containing the parameters for performing an open-ended analysis.
requestOptionsRequestOptionsNoRequest-specific configuration.

The AnalyzeRequest class contains the following properties:

NameTypeRequiredDescription
videoIdstringYesThe unique identifier of the video you wish to analyze and generate text for.
promptstringYesThe prompt to customize the output. The maximum length is 2,000 tokens.
temperaturenumberNoControls the randomness of the text output. Higher values generate more creative text, while lower values produce more deterministic output. Default: 0.2, Min: 0, Max: 1.
responseFormatTwelvelabsApi.ResponseFormatNoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text. This parameter is only valid when type is set to summary.
max_tokensnumberNoThe maximum number of tokens to generate. Min: 1. Max: 4096.

The ResponseFormat class contains the following properties:

NameTypeDescription
typeTwelvelabsApi.ResponseFormatTypeSet this parameter to “json_schema” to receive structured JSON responses.
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value

Returns a Promise that resolves to a NonStreamAnalyzeResponse object containing the generated text.

The NonStreamAnalyzeResponse interface contains the following properties:

NameTypeDescription
idstringUnique identifier of the response.
datastringThe generated text based on the prompt you provided.
finishReasonTwelvelabsApi.FinishReasonThe reason for the generation to finish. Values:
- null: The generation hasn’t finished yet.
- stop: The generation stopped because the model reached the maximum number of tokens. For JSON responses, this may result in truncated, invalid JSON that fails to parse.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.

The maximum length of the response is 4,096 tokens.

API Reference

Open-ended analysis page.

Analyze videos.

Open-ended analysis with streaming responses

Description: This method analyzes a video and generates text based on its content.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1async analyzeStream(
2  request: TwelvelabsApi.AnalyzeStreamRequest,
3  requestOptions?: TwelvelabsApiClient.RequestOptions
4): core.HttpResponsePromise<core.Stream<TwelvelabsApi.StreamAnalyzeResponse>>

Parameters

NameTypeRequiredDescription
requestTwelvelabsApi.AnalyzeStreamRequestYesThe request object containing the parameters for performing an open-ended analysis.
requestOptions[RequestOptions]NoRequest-specific configuration.

The AnalyzeStreamRequest class contains the following properties:

NameTypeRequiredDescription
videoIdstringYesThe unique identifier of the video you wish to analyze and generate text for.
promptstringYesThe prompt to customize the output. The maximum length is 2,000 tokens.
temperaturenumberNoControls the randomness of the text output. Higher values generate more creative text, while lower values produce more deterministic output. Default: 0.2, Min: 0, Max: 1.
responseFormatTwelvelabsApi.ResponseFormatNoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text. This parameter is only valid when type is set to summary.
max_tokensnumberNoThe maximum number of tokens to generate. Min: 1. Max: 4096.

The ResponseFormat class contains the following properties:

NameTypeDescription
typeTwelvelabsApi.ResponseFormatTypeSet this parameter to “json_schema” to receive structured JSON responses.
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value

Returns a promise that resolves to a Stream<TwelvelabsApi.StreamAnalyzeResponse> object that can be iterated over to receive streaming text chunks.

The StreamAnalyzeResponse can be either a StreamAnalyzeResponse.StreamStart, a StreamAnalyzeResponse.TextGeneration, or a StreamAnalyzeResponse.StreamEnd.

The StreamAnalyzeResponse.StreamStart interface contains the following properties:

NameTypeDescription
eventTypestringThis field is always set to stream_start for this event.
metadataStreamStartResponseMetadataAn object containing metadata about the stream.

The StreamAnalyzeResponse.TextGeneration interface contains the following properties:

NameTypeDescription
eventTypestringThis field is always set to text_generation for this event.
textstringA fragment of the generated text. Note that text fragments may be split at arbitrary points, not necessarily at word or sentence boundaries.

The StreamAnalyzeResponse.StreamEnd interface contains the following properties:

NameTypeDescription
eventTypestringThis field is always set to stream_end for this event.
metadataStreamEndResponseMetadataAn object containing metadata about the stream.
finishReasonTwelvelabsApi.FinishReasonThe reason for the generation to finish. Values:
- null: The generation hasn’t finished yet.
- stop: The generation stopped because the model reached the maximum number of tokens. For JSON responses, this may result in truncated, invalid JSON that fails to parse.

The StreamStartResponseMetadata interface contains the following properties:

NameTypeDescription
generationIdstringA unique identifier for the generation session.

The StreamEndResponseMetadata interface contains the following properties:

NameTypeDescription
generationIdstringThe same unique identifier provided in the stream_start event.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.

The maximum length of the response is 4,096 tokens.

API Reference

Open-ended analysis page.

Analyze videos.

Error codes

This section lists the most common error messages you may encounter while analyzing videos.

  • token_limit_exceeded
    • Your request could not be processed due to exceeding maximum token limit. Please try with another request or another video with shorter duration.
  • index_not_supported_for_generate
    • You can only summarize videos uploaded to an index with an engine from the Pegasus family enabled.