For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sample appsIntegrationsDiscordPlaygroundDevEx repo
GuidesSDK ReferenceAPI Reference
GuidesSDK ReferenceAPI Reference
  • Python SDK
    • The TwelveLabs class
    • Manage indexes
    • Upload content
    • Index content
    • Manage videos
    • Manage entities
    • Search
    • Create embeddings v2
    • Create embeddings v1
    • Analyze and segment videos
  • Node.js SDK
    • The TwelveLabs class
    • Manage indexes
    • Upload content
    • Index content
    • Manage videos
    • Manage entities
    • Search
    • Create embeddings v2
    • Create embeddings v1
    • Analyze and segment videos
      • Sync analysis
      • Async analysis
LogoLogo
Sample appsIntegrationsDiscordPlaygroundDevEx repo
On this page
  • Methods
  • List analysis tasks
  • Create an async analysis task
  • VideoContext
  • AnalyzePromptV2
  • SegmentDefinition
  • SegmentField
  • SmeMediaSource
  • Retrieve task status and results
  • Delete an analysis task
Node.js SDKAnalyze and segment videos

Async analysis

Was this page helpful?
Previous
Built with

The AnalyzeAsync.Tasks interface provides methods to analyze videos asynchronously, generate text, and extract structured, timestamped segments. The platform supports two analysis modes: general analysis (prompt-based text generation) with both Pegasus 1.2 and Pegasus 1.5, and video segmentation with Pegasus 1.5.

When to use this interface:

  • Generate custom text from your video using a prompt (Pegasus 1.2 only)
  • Extract timestamped metadata with custom fields from your video (Pegasus 1.5 only)
  • Analyze videos longer than 1 hour
  • Process videos up to 2 hours
  • Avoid blocking your application during video processing

Do not use this interface for:

  • Videos under 1 hour for which you need immediate results or real-time streaming. Use the analyze method instead.
Input requirements
  • Minimum duration: 4 seconds
  • Maximum duration: 2 hours
  • Formats: FFmpeg supported formats
  • Resolution: 360x360 to 5184x2160 pixels
  • Aspect ratio: Between 1:1 and 1:2.4, or between 2.4:1 and 1:1

On the Free plan, analysis hours count toward a shared limit that also covers indexing - the number of segment definitions does not affect this limit. On paid plans, you pay based on how much video you process and how many segment definitions you include - see the Frequently asked questions page for examples.

Analyzing videos asynchronously requires three steps:

  1. Create an analysis task using the create method. The platform returns a task ID.
  2. Poll the status of the task using the retrieve method. Wait until the status is ready.
  3. Retrieve the results from the response when the status is ready using the retrieve method.

Methods

List analysis tasks

Description: This method returns a list of the analysis tasks in your account. The platform returns your analysis tasks sorted by creation date, with the newest at the top of the list.

Function signature and example:

1list(
2    request?: TwelvelabsApi.analyzeAsync.TasksListRequest,
3    requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.analyzeAsync.TasksListResponse>

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.analyzeAsync.TasksListRequestNoParameters for listing analysis tasks.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

The TwelvelabsApi.analyzeAsync.TasksListRequest interface has the following properties:

NameTypeRequiredDescription
pagenumberNoA number that identifies the page to retrieve. Default: 1.
pageLimitnumberNoThe number of items to return on each page. Default: 10. Max: 50.
statusTwelvelabsApi.AnalyzeTaskStatusNoFilter analysis tasks by status. Values: queued, pending, processing, ready, failed.
videoUrlstringNoFilter tasks by exact video source URL.
assetIdstringNoFilter tasks by asset ID.
videoIdstringNoFilter tasks by video ID for pre-indexed videos (Pegasus 1.2 only). Deprecated — use assetId instead.
analysisModeTwelvelabsApi.analyzeAsync.TasksListRequestAnalysisModeNoFilter tasks by the analysis mode used when creating the task. Values: "general", "time_based_metadata".

Return value: Returns an HttpResponsePromise that resolves to a TwelvelabsApi.analyzeAsync.TasksListResponse object.

The TwelvelabsApi.analyzeAsync.TasksListResponse interface contains the following properties:

NameTypeDescription
dataTwelvelabsApi.AnalyzeTaskResponse[]An array that contains up to pageLimit analysis tasks.
pageInfoTwelvelabsApi.PageInfoAn object that provides information about pagination.

The TwelvelabsApi.PageInfo interface contains the following properties:

NameTypeDescription
limitPerPagenumberThe number of items returned per page.
pagenumberThe current page number.
totalPagenumberThe total number of pages.
totalResultsnumberThe total number of analysis tasks in your account.

For details about AnalyzeTaskResponse, see Retrieve task status and results.

API Reference: List async analysis tasks

Create an async analysis task

Description: This method asynchronously analyzes your videos. It supports two modes: general analysis (prompt-based text generation) with both Pegasus 1.2 and Pegasus 1.5, and video segmentation with Pegasus 1.5.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1create(
2    request: TwelvelabsApi.analyzeAsync.CreateAsyncAnalyzeRequest,
3    requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.CreateAnalyzeTaskResponse>

Parameters:

NameTypeRequiredDescription
requestTwelvelabsApi.analyzeAsync.CreateAsyncAnalyzeRequestYesParameters for creating an async analysis task.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

The TwelvelabsApi.analyzeAsync.CreateAsyncAnalyzeRequest interface has the following properties:

NameTypeRequiredDescription
modelNameTwelvelabsApi.analyzeAsync.CreateAsyncAnalyzeRequestModelNameNoThe video understanding model to use for analysis. Values:
- "pegasus1.2": General analysis (prompt-based text generation).
- "pegasus1.5": General analysis with video clipping, structured prompts, longer responses, and video segmentation. See the Pegasus page for details.
Default: "pegasus1.2"
customIdstringNoAn optional identifier that you set when you create the task. Use this field to correlate tasks across responses, for example, to distinguish tasks by type or environment. Must match the pattern ^[a-zA-Z0-9_-]{1,64}$.
videoTwelvelabsApi.VideoContextYesAn object that specifies the source of the video. See VideoContext for details.
promptstringNoA text prompt that guides the model on the desired format or content. Works with both Pegasus 1.2 and Pegasus 1.5. To include reference images in your prompt, use the promptV2 parameter instead (Pegasus 1.5 only). Mutually exclusive with the promptV2 parameter.
promptV2TwelvelabsApi.AnalyzePromptV2NoA structured prompt with <@name> placeholders for referencing images. Requires the modelName parameter set to "pegasus1.5". Not supported when the analysisMode parameter is "time_based_metadata". Mutually exclusive with the prompt parameter. See AnalyzePromptV2.
analysisModeTwelvelabsApi.analyzeAsync.CreateAsyncAnalyzeRequestAnalysisModeNoThe analysis pipeline to use. Value: "time_based_metadata" (segments the video into time-based intervals and extracts metadata for each segment). Requires modelName set to "pegasus1.5" and responseFormat.type set to "segment_definitions".
temperaturenumberNoControls the randomness of the text output. Default: 0.2, Min: 0, Max: 1
maxTokensnumberNoThe maximum response length, in tokens. For "pegasus1.2": Min: 1, Max: 4,096. For "pegasus1.5" general mode: Min: 512, Max: 98,304, Default: 4,096. For "pegasus1.5" time_based_metadata mode: Min: 2,048, Max: 98,304, Default: 32,768. For Pegasus 1.5, the input and response must fit within the context window.
responseFormatTwelvelabsApi.AsyncResponseFormatNoControls the response format. When you omit this parameter, you receive unstructured text.
minSegmentDurationnumberNoMinimum duration for each extracted segment, in seconds. Prevents the model from creating very short segments. Requires modelName set to "pegasus1.5" and analysisMode set to "time_based_metadata". Min: 2.
maxSegmentDurationnumberNoMaximum duration for each extracted segment, in seconds. Breaks long continuous sections into shorter segments. Must be greater than or equal to minSegmentDuration. Requires modelName set to "pegasus1.5" and analysisMode set to "time_based_metadata". Min: 2.
startTimenumberNoStart of the analysis window, in seconds. Use with endTime to analyze only a portion of the video. Requires modelName set to "pegasus1.5". If omitted, defaults to 0. The clip (endTime − startTime) must be at least 4 seconds. Mutually exclusive with responseFormat.segmentDefinitions[].timeRanges.
endTimenumberNoEnd of the analysis window, in seconds. Use with startTime to analyze only a portion of the video. Requires modelName set to "pegasus1.5". If omitted, defaults to the video duration. The clip (endTime − startTime) must be at least 4 seconds. Mutually exclusive with responseFormat.segmentDefinitions[].timeRanges.

VideoContext

The TwelvelabsApi.VideoContext type specifies the source of the video. Provide exactly one of the following:

TypeFieldTypeDescription
VideoContext.UrlurlstringThe publicly accessible URL of the video file. Use direct links to raw media files. Video hosting platforms and cloud storage sharing links are not supported.
VideoContext.AssetIdassetIdstringThe unique identifier of an asset from a direct or multipart upload. The asset status must be ready. Use assets.retrieve to check the status.
VideoContext.Base64Stringbase64StringstringThe base64-encoded video data. The maximum size is 30 MB.

Set the type discriminator field to "url", "asset_id", or "base64_string" to indicate which source you are providing.

The AsyncResponseFormat class contains the following properties:

NameTypeDescription
typeTwelvelabsApi.AsyncResponseFormatTypeThe response format to use. Values: "json_schema" (structured JSON conforming to a provided schema), "segment_definitions" (timestamped metadata with custom fields, requires modelName set to "pegasus1.5" and analysisMode set to "time_based_metadata").
jsonSchemaRecord<string, unknown>Contains the JSON schema that defines the response structure. The schema must adhere to the JSON Schema Draft 2020-12 specification. For details, see the json_schema parameter in the API Reference section.
segmentDefinitionsTwelvelabsApi.SegmentDefinition[]Define the types of segments to extract from your video. Required when type is "segment_definitions". Minimum 1, maximum 10 definitions. See SegmentDefinition for details.

AnalyzePromptV2

The AnalyzePromptV2 interface defines a structured prompt with image references.

NameTypeRequiredDescription
inputTextstringYesThe text of the prompt. Use <@name> placeholders to reference images declared in mediaSources (Example: "Is there a <@tiger-1> in the video?"). For Pegasus 1.5, this text counts toward the context window.
mediaSourcesTwelvelabsApi.SmeMediaSource[]NoReference images for the <@name> placeholders in the prompt. Maximum 4 sources. See SmeMediaSource.

SegmentDefinition

The TwelvelabsApi.SegmentDefinition interface defines a type of segment to extract from the video.

NameTypeRequiredDescription
idstringYesA unique identifier for this segment definition.
descriptionstringYesDescribe what this type of segment looks like in the video. The model uses this text to identify matching segments.
fieldsTwelvelabsApi.SegmentField[]NoCustom fields to extract for each segment instance. Maximum 20 fields. See SegmentField for details.
mediaSourcesTwelvelabsApi.SmeMediaSource[]NoReference images that help the model identify segments. Maximum 4 sources. See SmeMediaSource for details.

SegmentField

The TwelvelabsApi.SegmentField interface defines a custom field to extract for each segment.

NameTypeRequiredDescription
namestringYesThe name of the field.
typeTwelvelabsApi.SegmentFieldTypeYesThe data type of the field. Values: "string", "boolean", "number", "integer", "array".
descriptionstringNoInstructions that guide the model on what this field should contain and how to extract it from the video.
enumstring[]NoAllowed values for this field. Maximum 50 values.
itemsTwelvelabsApi.SegmentFieldItemsNoRequired when type is "array". Specifies the type of array elements. The items object has a single property type with values: "string", "number", "boolean", "integer".

SmeMediaSource

The TwelvelabsApi.SmeMediaSource interface defines a reference image that provides visual context for segment identification. Provide exactly one of url, assetId, or base64String.

NameTypeRequiredDescription
namestringYesA descriptive name for this media source.
mediaTypeTwelvelabsApi.SmeMediaSourceMediaTypeYesThe media type. Value: "image".
urlstringNoA publicly accessible HTTPS URL of the image.
assetIdstringNoThe unique identifier of an uploaded asset.
base64StringstringNoBase64-encoded image data. The maximum size is 30MB.

Return value: Returns an HttpResponsePromise that resolves to a TwelvelabsApi.CreateAnalyzeTaskResponse object containing the task details.

The TwelvelabsApi.CreateAnalyzeTaskResponse interface contains the following properties:

NameTypeDescription
taskIdstringThe unique identifier of the analysis task.
statusTwelvelabsApi.AnalyzeTaskStatusThe initial status of the task. Value: queued.

API Reference: Create an async analysis task

Retrieve task status and results

Description: This method retrieves the status and results of an analysis task.

Task statuses:

  • queued: The task is waiting to be processed.
  • pending: The task is queued and waiting to start.
  • processing: The platform is analyzing the video.
  • ready: Processing is complete. Results are available in the response.
  • failed: The task failed. No results were generated.

Call this method repeatedly until status is ready or failed. When status is ready, use the results from the response.

Function signature and example:

1retrieve(
2    taskId: string,
3    requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.AnalyzeTaskResponse>

Parameters:

NameTypeRequiredDescription
taskIdstringYesThe unique identifier of the analysis task.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns an HttpResponsePromise that resolves to a TwelvelabsApi.AnalyzeTaskResponse object containing the task status and results.

The TwelvelabsApi.AnalyzeTaskResponse interface contains the following properties:

NameTypeDescription
taskIdstringThe unique identifier of the analysis task.
videoSourceTwelvelabsApi.AnalyzeTaskResponseVideoSourceThe video source you provided. Only present for tasks that use direct video input (url, base64String, or assetId).
requestParamsTwelvelabsApi.AnalyzeTaskResponseRequestParamsThe parameters you sent when creating this task. Only present for tasks created with modelName set to "pegasus1.5".
statusTwelvelabsApi.AnalyzeTaskStatusThe current status of the task. Values: queued, pending, processing, ready, failed.
createdAtDateThe date and time when the task was created, in RFC 3339 format.
completedAtDateThe date and time when the task completed or failed, in RFC 3339 format. The platform returns this field only when status is ready or failed.
resultTwelvelabsApi.AnalyzeTaskResultAn object that contains the generated text and additional information. The platform returns this object only when status is ready.
errorTwelvelabsApi.AnalyzeTaskErrorA message attached to the task response. The platform sets this field in two cases:
- status is "failed": The message field describes the failure.
- status is "ready" and result.finishReason is "length": The message field contains a truncation warning. The truncated output is in result.data.
Not set when status is "ready" and result.finishReason is "stop".
webhooksTwelvelabsApi.AnalyzeTaskWebhookInfo[]The delivery status of each webhook endpoint. The platform omits this field when there are no webhooks configured.

The TwelvelabsApi.AnalyzeTaskResponseVideoSource interface contains the following properties:

NameTypeDescription
typeTwelvelabsApi.AnalyzeTaskResponseVideoSourceTypeThe type of video source. Values: "url", "base64_string", "asset_id", "video_id".
urlstringThe video URL. Present when type is "url".
assetIdstringThe asset ID. Present when type is "asset_id".
videoIdstringThe video ID. Present when type is "video_id". Deprecated — use assetId instead.
systemMetadataTwelvelabsApi.AnalyzeTaskResponseVideoSourceSystemMetadataSystem-extracted video metadata. Present on a best-effort basis once the video has been processed.

The TwelvelabsApi.AnalyzeTaskResponseVideoSourceSystemMetadata interface contains the following properties:

NameTypeDescription
durationnumberThe video duration in seconds.

The TwelvelabsApi.AnalyzeTaskResponseRequestParams interface contains the following properties:

NameTypeDescription
analysisModeTwelvelabsApi.AnalyzeTaskResponseRequestParamsAnalysisModeThe analysis pipeline used for this task. Values: "general", "time_based_metadata".
responseFormatTwelvelabsApi.AnalyzeTaskResponseRequestParamsResponseFormatThe response format you configured. Present only when you included it in the request.
temperaturenumberThe temperature value used for the analysis.
maxTokensnumberThe maximum response length you set, in tokens.
minSegmentDurationnumberThe minimum segment duration you set, in seconds. Present when analysisMode is "time_based_metadata".
maxSegmentDurationnumberThe maximum segment duration you set, in seconds. Present when analysisMode is "time_based_metadata".

The TwelvelabsApi.AnalyzeTaskResult interface contains the following properties:

NameTypeDescription
generationIdstringThe unique identifier for the generation session.
datastringThe generated text for this analysis task. When analysisMode is not set, a plain-text string based on the prompt you provided. When analysisMode is "time_based_metadata", a JSON-encoded string keyed by segment definition id. Each key maps to an array of segment objects with start_time (number), end_time (number), and metadata (object with your custom fields).
finishReasonTwelvelabsApi.FinishReasonThe reason the generation stopped. Values: null (generation has not finished), "stop" (the generation reached the end of the output text), "length" (generation stopped early because the response reached the maximum response length or the context window; the partial output is in data, and a warning is in the task’s error field).
usageTwelvelabsApi.AnalyzeTaskResultUsageThe number of tokens used in the generation.

The TwelvelabsApi.AnalyzeTaskResultUsage interface contains the following properties:

NameTypeDescription
outputTokensnumberThe number of tokens in the generated text.
inputTokensnumberThe number of tokens the input consumed. Together with outputTokens, this value must fit within the context window.

The TwelvelabsApi.AnalyzeTaskError interface contains the following properties:

NameTypeDescription
messagestringA message that describes a task failure or truncation warning. Check finishReason instead of parsing the message text. For truncation messages, see Error codes.

API Reference: Retrieve analysis task status and results

Delete an analysis task

Description: This method deletes an analysis task. You can only delete tasks that are not currently being processed.

Function signature and example:

1delete(
2    taskId: string,
3    requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<void>

Parameters:

NameTypeRequiredDescription
taskIdstringYesThe unique identifier of the analysis task.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns an HttpResponsePromise that resolves to void. If successful, the platform returns a 204 No Content response.

API Reference: Delete an analysis task