Analyze videos

The TwelveLabs Node.js SDK provides methods to analyze videos to generate text from their content.

Titles, topics, and hashtags

Description: This method analyzes a specific video and generates titles, topics, and hashtags based on its content. It uses predefined formats and doesn’t require a custom prompt, and it’s best for generating immediate and straightforward text representations without specific customization.

Function signature and example:

1async gist(
2 request: TwelvelabsApi.GistRequest,
3 requestOptions?: TwelvelabsApiClient.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.Gist>

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.GistRequestYesThe request object containing the video ID and gist types.
requestOptionsTwelvelabsApiClient.RequestOptionsNoRequest-specific configuration.

The GistRequest interface contains the following properties:

NameTypeRequiredDescription
videoIdstringYesThe unique identifier of the video for which you want to generate text.
typesTwelvelabsApi.GistRequestTypesItem[]YesThe types of text you want to generate. Available values: "title", "topic", "hashtag".

Return value: Returns an HttpResponsePromise that resolves to a TwelvelabsApi.Gist instance.

The Gist interface contains the following properties:

NameTypeDescription
idstringUnique identifier of the response.
titlestringSuggested title for the video.
topicsstring[]An array of topics that are relevant to the video.
hashtagsstring[]An array of hashtags that are relevant to the video.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.

API Reference: Titles, topics, and hashtags page.

Related guide: Titles, topics, and hashtags.

Summaries, chapters, and highlights

Description: This method analyzes a video and generates summaries, chapters, or highlights based on its content. Optionally, you can provide a prompt to customize the output.

Function signature and example:

1summarize(
2 request: TwelvelabsApi.SummarizeRequest,
3 requestOptions?: TwelvelabsApiClient.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.SummarizeResponse>;

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.SummarizeRequestYesSpecifies the type of text you wish to generate.
request_optionsTwelvelabsApiClient.RequestOptionsNoRequest-specific configuration.

The SummarizeRequest interface contains the following properties:

NameTypeRequiredDescription
videoIdstringYesThe unique identifier of the video that you want to summarize.
typestringYesSpecifies the type of text. Use summary for a brief report, chapter for a chronological list of chapters, or highlight for key moments.
promptstringNoUse this field to provide context for the summarization task, such as the target audience, style, tone of voice, and purpose. Maximum length is 2,000 tokens.
temperaturenumberNoControls the randomness of the text output. A higher value generates more creative text, while a lower value produces more deterministic text. Default: 0.2, Min: 0, Max: 1.
responseFormatTwelvelabsApi.ResponseFormatNoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text. This parameter is only valid when type is set to summary.
max_tokensnumberNoThe maximum number of tokens to generate. Min: 1. Max: 4096.

The ResponseFormat class contains the following properties:

NameTypeDescription
typestringSet this parameter to “json_schema” to receive structured JSON responses.
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value: Returns a Promise that resolves to a SummarizeResponse object containing the generated content. The response type varies based on the type parameter.

When type is "summary": Returns a SummarizeSummaryResult object with the following properties:

NameTypeDescription
summarizeType"summary"Indicates this is a summary response.
idstringUnique identifier of the response.
summarystringA brief report of the main points of the video.
usageTokenUsageThe number of tokens used in the generation.

When type is "chapter": Returns a SummarizeChapterResult object with the following properties:

NameTypeDescription
summarizeType"chapter"Indicates this is a chapter response.
idstringUnique identifier of the response.
chaptersSummarizeChapterResultChaptersItem[]An array that contains details about the detected chapters and their content.
usageTokenUsageThe number of tokens used in the generation.

When type is "highlight": Returns a SummarizeHighlightResult object with the following properties:

NameTypeDescription
summarizeType"highlight"Indicates this is a highlight response.
idstringUnique identifier of the response.
highlightsSummarizeHighlightResultHighlightsItem[]An array that contains the highlights.
usageTokenUsageThe number of tokens used in the generation.

The SummarizeChapterResultChaptersItem interface contains the following properties:

NameTypeDescription
chapterNumbernumberThe sequence number of the chapter. Note that this field starts at 0.
startSecnumberThe starting time of the chapter, measured in seconds from the beginning of the video.
endSecnumberThe ending time of the chapter, measured in seconds from the beginning of the video.
chapterTitlestringThe title of the chapter.
chapterSummarystringA brief summary describing the content of the chapter.

The SummarizeHighlightResultHighlightsItem interface contains the following properties:

NameTypeDescription
startSecnumberThe starting time of the highlight, measured in seconds from the beginning of the video.
endSecnumberThe ending time of the highlight, measured in seconds from the beginning of the video.
highlightstringThe title of the highlight.
highlightSummarystringA brief description that captures the essence of this part of the video.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.

API Reference: Summaries, chapters, and highlights.

Related guide: Summaries, chapters, and highlights.

Open-ended analysis

Description: This method analyzes a video and generates text based on its content.

Function signature and example:

1analyze(request: TwelvelabsApi.AnalyzeRequest, requestOptions?: TwelvelabsApiClient.RequestOptions): core.HttpResponsePromise<TwelvelabsApi.NonStreamAnalyzeResponse>;

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.AnalyzeRequestYesThe request object containing the parameters for performing an open-ended analysis.
requestOptionsRequestOptionsNoRequest-specific configuration.

The AnalyzeRequest class contains the following properties:

NameTypeRequiredDescription
videoIdstringYesThe unique identifier of the video you wish to analyze and generate text for.
promptstringYesThe prompt to customize the output. The maximum length is 2,000 tokens.
temperaturenumberNoControls the randomness of the text output. Higher values generate more creative text, while lower values produce more deterministic output. Default: 0.2, Min: 0, Max: 1.
response_formattyping.Optional[ResponseFormat]NoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text.
max_tokenstyping.Optional[int]NoThe maximum number of tokens to generate. Min: 1. Max: 4096.

Return value: Returns a Promise that resolves to a NonStreamAnalyzeResponse object containing the generated text.

The NonStreamAnalyzeResponse interface contains the following properties:

NameTypeDescription
idstringUnique identifier of the response.
datastringThe generated text based on the prompt you provided.
finishReasonTwelvelabsApi.FinishReasonThe reason for the generation to finish. Values:
- null: The generation hasn’t finished yet.
- stop: The generation stopped because the model reached the maximum number of tokens. For JSON responses, this may result in truncated, invalid JSON that fails to parse.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.

The maximum length of the response is 4,096 tokens.

API Reference: Open-ended analysis page.

Related guide: Open-ended analysis.

Open-ended analysis with streaming responses

Description: This method analyzes a video and generates text based on its content.

Function signature and example:

1async analyzeStream(
2 request: TwelvelabsApi.AnalyzeStreamRequest,
3 requestOptions?: TwelvelabsApiClient.RequestOptions
4): core.HttpResponsePromise<core.Stream<TwelvelabsApi.StreamAnalyzeResponse>>

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.AnalyzeStreamRequestYesThe request object containing the parameters for performing an open-ended analysis.
requestOptions[RequestOptions]NoRequest-specific configuration.

The AnalyzeStreamRequest class contains the following properties:

NameTypeRequiredDescription
videoIdstringYesThe unique identifier of the video you wish to analyze and generate text for.
promptstringYesThe prompt to customize the output. The maximum length is 2,000 tokens.
temperaturenumberNoControls the randomness of the text output. Higher values generate more creative text, while lower values produce more deterministic output. Default: 0.2, Min: 0, Max: 1.
responseFormatTwelvelabsApi.ResponseFormatNoSpecifies the format of the response. If you omit this parameter, the platform returns unstructured text. This parameter is only valid when type is set to summary.
max_tokensnumberNoThe maximum number of tokens to generate. Min: 1. Max: 4096.

The ResponseFormat class contains the following properties:

NameTypeDescription
typestringSet this parameter to “json_schema” to receive structured JSON responses.
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value: Returns a promise that resolves to a Stream<TwelvelabsApi.StreamAnalyzeResponse> object that can be iterated over to receive streaming text chunks.

The StreamAnalyzeResponse can be either a StreamAnalyzeResponse.StreamStart, a StreamAnalyzeResponse.TextGeneration, or a StreamAnalyzeResponse.StreamEnd.

The StreamAnalyzeResponse.StreamStart interface contains the following properties:

NameTypeDescription
eventTypestringThis field is always set to stream_start for this event.
metadataStreamStartResponseMetadataAn object containing metadata about the stream.

The StreamAnalyzeResponse.TextGeneration interface contains the following properties:

NameTypeDescription
eventTypestringThis field is always set to text_generation for this event.
textstringA fragment of the generated text. Note that text fragments may be split at arbitrary points, not necessarily at word or sentence boundaries.

The StreamAnalyzeResponse.StreamEnd interface contains the following properties:

NameTypeDescription
eventTypestringThis field is always set to stream_end for this event.
metadataStreamEndResponseMetadataAn object containing metadata about the stream.
finishReasonTwelvelabsApi.FinishReasonThe reason for the generation to finish. Values:
- null: The generation hasn’t finished yet.
- stop: The generation stopped because the model reached the maximum number of tokens. For JSON responses, this may result in truncated, invalid JSON that fails to parse.

The StreamStartResponseMetadata interface contains the following properties:

NameTypeDescription
generationIdstringA unique identifier for the generation session.

The StreamEndResponseMetadata interface contains the following properties:

NameTypeDescription
generationIdstringThe same unique identifier provided in the stream_start event.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.

The maximum length of the response is 4,096 tokens.

API Reference: Open-ended analysis page.

Related guide: Open-ended analysis.

Error codes

This section lists the most common error messages you may encounter while analyzing videos.

  • token_limit_exceeded
    • Your request could not be processed due to exceeding maximum token limit. Please try with another request or another video with shorter duration.
  • index_not_supported_for_generate
    • You can only summarize videos uploaded to an index with an engine from the Pegasus family enabled.