For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sample appsIntegrationsDiscordPlaygroundDevEx repo
GuidesSDK ReferenceAPI Reference
GuidesSDK ReferenceAPI Reference
  • Python SDK
    • The TwelveLabs class
    • Manage indexes
    • Upload content
    • Index content
    • Manage videos
    • Manage entities
    • Search
    • Create embeddings v2
    • Create embeddings v1
    • Analyze and segment videos
  • Node.js SDK
    • The TwelveLabs class
    • Manage indexes
    • Upload content
    • Index content
    • Manage videos
    • Manage entities
    • Search
    • Create embeddings v2
    • Create embeddings v1
    • Analyze and segment videos
      • Sync analysis
      • Async analysis
LogoLogo
Sample appsIntegrationsDiscordPlaygroundDevEx repo
On this page
  • Sync analysis
  • VideoContext
  • AnalyzePromptV2
  • SmeMediaSource
  • Sync analysis with streaming responses
  • Error codes
Node.js SDKAnalyze and segment videos

Sync analysis

Was this page helpful?
Previous

Async analysis

Next
Built with

The TwelvelabsApiClient interface provides methods to analyze videos synchronously and generate text based on their content. These methods return results immediately in the response.

When to use these methods:

  • Analyze videos up to 1 hour
  • Retrieve immediate results without polling for task completion
  • Stream text fragments in real time for immediate processing and feedback

Do not use these methods for:

  • Videos longer than 1 hour. Use the analyzeAsync.tasks.create method instead.
  • Video segmentation with custom segment definitions. Use the analyzeAsync.tasks.create method with modelName set to "pegasus1.5" instead.

Sync analysis

Description: This method analyzes your videos and returns the results directly in the response. It generates text based on your prompts and supports both Pegasus 1.2 and Pegasus 1.5 for general analysis (prompt-based text generation).

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1analyze(request: TwelvelabsApi.AnalyzeRequest, requestOptions?: TwelvelabsApiClient.RequestOptions): core.HttpResponsePromise<TwelvelabsApi.NonStreamAnalyzeResponse>;

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.AnalyzeRequestYesThe request object containing the parameters for performing an analysis.
requestOptionsRequestOptionsNoRequest-specific configuration.

The AnalyzeRequest interface contains the following properties:

NameTypeRequiredDescription
modelNamestringNoThe video understanding model to use for analysis. Values:
- "pegasus1.2": General analysis (prompt-based text generation).
- "pegasus1.5": General analysis with video clipping, structured prompts, longer responses, and video segmentation (async only). See the Pegasus page for details.
Default: "pegasus1.2"
videoIdstringNoThe unique identifier of the video to analyze. Use this parameter when the modelName parameter is "pegasus1.2". Not supported with "pegasus1.5". This parameter will be deprecated and removed in a future version. Use the video parameter instead.
videoTwelvelabsApi.VideoContextNoAn object specifying the source of the video content. Include exactly one source. See VideoContext.
promptstringNoA text prompt that guides the model on the desired format or content. Works with both Pegasus 1.2 and Pegasus 1.5. To include reference images in your prompt, use the promptV2 parameter instead (Pegasus 1.5 only). Mutually exclusive with the promptV2 parameter.
promptV2TwelvelabsApi.AnalyzePromptV2NoA structured prompt with <@name> placeholders for referencing images. Requires the modelName parameter set to "pegasus1.5". Mutually exclusive with the prompt parameter. See AnalyzePromptV2.
temperaturenumberNoControls the randomness of the text output. Default: 0.2, Min: 0, Max: 1
startTimenumberNoStart of the analysis window, in seconds. Use with endTime to analyze only a portion of the video. Requires modelName set to "pegasus1.5". If omitted, defaults to 0. Must be less than endTime and less than the video duration. The clip (endTime − startTime) must be at least 4 seconds.
endTimenumberNoEnd of the analysis window, in seconds. Use with startTime to analyze only a portion of the video. Requires modelName set to "pegasus1.5". If omitted, defaults to the video duration. Must be greater than startTime and less than or equal to the video duration. The clip (endTime − startTime) must be at least 4 seconds.

| responseFormat | TwelvelabsApi.SyncResponseFormat | No | Specifies the format of the response. When you omit this parameter, the platform returns unstructured text. Only the json_schema type is supported for synchronous analysis. | | maxTokens | number | No | The maximum response length, in tokens. The allowed range depends on the model: Pegasus 1.2 — Min: 1, Max: 4,096, Default: 4,096. Pegasus 1.5 — Min: 512, Max: 98,304, Default: 4,096. For Pegasus 1.5, the input and response must fit within the context window. |

The SyncResponseFormat class contains the following properties:

NameTypeDescription
typeTwelvelabsApi.SyncResponseFormatTypeThe response format to use. Value: "json_schema" (structured JSON conforming to a provided schema).
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

VideoContext

The VideoContext type specifies the source of the video content. Provide exactly one of the following. Set the type discriminator field to "url", "asset_id", or "base64_string" to indicate which source you are providing.

ObjectDescription
{ type: "url", url: string }The publicly accessible URL of the video file. Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported.
{ type: "asset_id", assetId: string }The unique identifier of an asset from a direct or multipart upload. The asset status must be ready. Use assets.retrieve to check the status.
{ type: "base64_string", base64String: string }The base64-encoded video data. The maximum size is 30MB.

AnalyzePromptV2

The AnalyzePromptV2 interface defines a structured prompt with image references.

NameTypeRequiredDescription
inputTextstringYesThe text of the prompt. Use <@name> placeholders to reference images declared in mediaSources (Example: "Is there a <@tiger-1> in the video?"). For Pegasus 1.5, this text counts toward the context window.
mediaSourcesTwelvelabsApi.SmeMediaSource[]NoReference images for the <@name> placeholders in the prompt. Maximum 4 sources. See SmeMediaSource.

SmeMediaSource

A reference image that provides visual context. Provide exactly one of url, assetId, or base64String.

NameTypeRequiredDescription
namestringYesA descriptive name for this media source.
mediaTypeTwelvelabsApi.SmeMediaSourceMediaTypeYesThe media type. Value: "image".
urlstringNoA publicly accessible HTTPS URL of the image.
assetIdstringNoThe unique identifier of an uploaded asset.
base64StringstringNoBase64-encoded image data. The maximum size is 30MB.

Return value: Returns a Promise that resolves to a NonStreamAnalyzeResponse object containing the generated text.

The NonStreamAnalyzeResponse interface contains the following properties:

NameTypeDescription
idstringUnique identifier of the response.
datastringThe generated text based on the prompt you provided.
finishReasonTwelvelabsApi.FinishReasonThe reason the generation stopped. Values:
- null: The generation hasn’t finished yet.
- "stop": The generation reached the end of the output text.
- "length": The response reached the maximum response length or the context window. The partial output is returned and a warning is in the error field. Both Pegasus 1.5 and Pegasus 1.2 report "length". For JSON responses, this may return truncated JSON that fails to parse.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.
inputTokensnumberThe number of tokens the input consumed. Together with outputTokens, this value must fit within the context window.

API Reference: Sync analysis.

Related guide: Analyze videos.

Sync analysis with streaming responses

Description: This method analyzes your videos and returns the results directly in the response. It generates text based on your prompts and supports both Pegasus 1.2 and Pegasus 1.5 for general analysis (prompt-based text generation).

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1async analyzeStream(
2  request: TwelvelabsApi.AnalyzeStreamRequest,
3  requestOptions?: TwelvelabsApiClient.RequestOptions
4): core.HttpResponsePromise<core.Stream<TwelvelabsApi.StreamAnalyzeResponse>>

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.AnalyzeStreamRequestYesThe request object containing the parameters for performing an analysis.
requestOptionsRequestOptionsNoRequest-specific configuration.

The AnalyzeStreamRequest interface contains the following properties:

NameTypeRequiredDescription
modelNamestringNoThe video understanding model to use for analysis. Values:
- "pegasus1.2": General analysis (prompt-based text generation).
- "pegasus1.5": General analysis with video clipping, structured prompts, longer responses, and video segmentation (async only). See the Pegasus page for details.
Default: "pegasus1.2"
videoIdstringNoThe unique identifier of the video to analyze. Use this parameter when the modelName parameter is "pegasus1.2". Not supported with "pegasus1.5". This parameter will be deprecated and removed in a future version. Use the video parameter instead.
videoTwelvelabsApi.VideoContextNoAn object specifying the source of the video content. Include exactly one source. See VideoContext.
promptstringNoA text prompt that guides the model on the desired format or content. Works with both Pegasus 1.2 and Pegasus 1.5. To include reference images in your prompt, use the promptV2 parameter instead (Pegasus 1.5 only). Mutually exclusive with the promptV2 parameter.
promptV2TwelvelabsApi.AnalyzePromptV2NoA structured prompt with <@name> placeholders for referencing images. Requires the modelName parameter set to "pegasus1.5". Mutually exclusive with the prompt parameter. See AnalyzePromptV2.
temperaturenumberNoControls the randomness of the text output. Default: 0.2, Min: 0, Max: 1
startTimenumberNoStart of the analysis window, in seconds. Use with endTime to analyze only a portion of the video. Requires modelName set to "pegasus1.5". If omitted, defaults to 0. Must be less than endTime and less than the video duration. The clip (endTime − startTime) must be at least 4 seconds.
endTimenumberNoEnd of the analysis window, in seconds. Use with startTime to analyze only a portion of the video. Requires modelName set to "pegasus1.5". If omitted, defaults to the video duration. Must be greater than startTime and less than or equal to the video duration. The clip (endTime − startTime) must be at least 4 seconds.

| responseFormat | TwelvelabsApi.SyncResponseFormat | No | Specifies the format of the response. When you omit this parameter, the platform returns unstructured text. Only the json_schema type is supported for synchronous analysis. | | maxTokens | number | No | The maximum response length, in tokens. The allowed range depends on the model: Pegasus 1.2 — Min: 1, Max: 4,096, Default: 4,096. Pegasus 1.5 — Min: 512, Max: 98,304, Default: 4,096. For Pegasus 1.5, the input and response must fit within the context window. |

For details about VideoContext, AnalyzePromptV2, and SmeMediaSource, see the Sync analysis section above.

The SyncResponseFormat class contains the following properties:

NameTypeDescription
typeTwelvelabsApi.SyncResponseFormatTypeThe response format to use. Value: "json_schema" (structured JSON conforming to a provided schema).
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.

Return value: Returns a promise that resolves to a Stream<TwelvelabsApi.StreamAnalyzeResponse> object that can be iterated over to receive streaming text chunks.

The StreamAnalyzeResponse can be either a StreamAnalyzeResponse.StreamStart, a StreamAnalyzeResponse.TextGeneration, or a StreamAnalyzeResponse.StreamEnd.

The StreamAnalyzeResponse.StreamStart interface contains the following properties:

NameTypeDescription
eventType"stream_start"This field is always set to stream_start for this event.
metadataStreamStartResponseMetadataAn object containing metadata about the stream.

The StreamAnalyzeResponse.TextGeneration interface contains the following properties:

NameTypeDescription
eventType"text_generation"This field is always set to text_generation for this event.
textstringA fragment of the generated text. Text fragments may be split at arbitrary points, not necessarily at word or sentence boundaries.

The StreamAnalyzeResponse.StreamEnd interface contains the following properties:

NameTypeDescription
eventType"stream_end"This field is always set to stream_end for this event.
finishReasonTwelvelabsApi.FinishReasonThe reason the generation stopped. Values:
- null: The generation hasn’t finished yet.
- "stop": The generation reached the end of the output text.
- "length": The response reached the maximum response length or the context window. The partial output is returned and a warning is in the error field. Both Pegasus 1.5 and Pegasus 1.2 report "length". For JSON responses, this may return truncated JSON that fails to parse.
metadataStreamEndResponseMetadataAn object containing metadata about the stream.

The StreamStartResponseMetadata interface contains the following properties:

NameTypeDescription
generationIdstringA unique identifier for the generation session.

The StreamEndResponseMetadata interface contains the following properties:

NameTypeDescription
generationIdstringThe same unique identifier provided in the stream_start event.
usageTokenUsageThe number of tokens used in the generation.

The TokenUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.
inputTokensnumberThe number of tokens the input consumed. Together with outputTokens, this value must fit within the context window.

API Reference: Sync analysis.

Related guide: Analyze videos.

Error codes

This section lists the most common error messages you may encounter while analyzing videos.

  • token_limit_exceeded
    • The request exceeds the context window and cannot be processed. Reduce the prompt length, use a shorter video, or lower the max_tokens value.
  • output truncated: the generation reached the configured max_tokens. The partial output is returned; raise max_tokens (up to 98304) if you need a longer response.
    • The response reached the maximum response length. The platform returns the partial output and sets finish_reason to "length". This message appears in the error.message field.
  • output truncated: combined input and output tokens reached the model's context limit. The partial output is returned; consider reducing input size (shorter prompt, smaller video clip, fewer media bindings) or lowering max_tokens.
    • The request reached the context window. The platform returns the partial output and sets finish_reason to "length". This message appears in the error.message field.
  • index_not_supported_for_generate
    • You can only summarize videos uploaded to an index with an engine from the Pegasus family enabled.