Multipart uploads

The Multipart Upload API is available as a limited, private preview. To request access to the preview version, contact us at sales@twelvelabs.io.

The MultipartUploadClient class provides methods to manage multipart upload sessions.

Utilize multipart uploads to upload large files by dividing them into smaller chunks. This approach enables reliable uploads of large files, particularly when processing chunks in parallel or resuming interrupted transfers. TwelveLabs recommends using this method for files larger than 200 MB, with a maximum size limit of 4 GB.

This method creates an asset that you can use in different workflows.

Workflow

1

Determine the total size of your file in bytes. You’ll need this value when creating the upload session.

2

Create an upload session: Call the multipart_upload.create method, providing details about your file, including its size. The response contains, among other information, the unique identifier of the asset, a list of upload URLs, and the size of each chunk in bytes.

3

Split your file: Use the chunk size from the response to divide your file into chunks of the specified size.

4

Upload chunks: Transfer each chunk to its designated presigned URL. You can upload the chunks in parallel for improved performance. Save the ETag from each upload response for progress reporting.

5

(Optional) Request additional URLs: Call the multipart_upload.get_additional_presigned_urls if you need URLs for remaining chunks or if existing URLs expire.

6

Report progress: Submit completed chunks via the multipart_upload.report_chunk_batch method in batches as chunks finish uploading. Use the ETag from each chunk upload as proof of successful transfer.

7

Confirm completion: The upload is complete when the multipart_upload.get_status mathod returns status: 'completed'. Use the asset ID from step 1 to perform additional operations on your uploaded video.

8

What you do next depends on your use case:

  • For creating embeddings (videos, audio, images): Use the asset ID with the Embed API v2.
  • For search and analysis (videos): Index an asset using the asset ID.

Methods

List incomplete uploads

Description: This method returns a list of all incomplete multipart upload sessions in your account.

Function signature and example:

1def list_incomplete_uploads(
2 self,
3 *,
4 page: typing.Optional[int] = None,
5 page_limit: typing.Optional[int] = None,
6 request_options: typing.Optional[RequestOptions] = None,
7) -> SyncPager[IncompleteUploadSummary]

Parameters:

NameTypeRequiredDescription
pageintNoA number that identifies the page to retrieve. Default: 1.
page_limitintNoThe number of items to return on each page. Default: 10. Max: 50.
request_optionsRequestOptionsNoRequest-specific configuration.

Return value: Returns a SyncPager[IncompleteUploadSummary] object that allows you to iterate through the paginated results.

The SyncPager[T] class contains the following properties and methods:

NameTypeDescription
itemsOptional[List[T]]A list containing the current page of items. Can be None.
has_nextboolIndicates whether there is a next page to load.
get_nextOptional[Callable[[], Optional[SyncPager[T]]]]A callable function that retrieves the next page. Can be None.
responseOptional[BaseHttpResponse]The HTTP response object. Can be None.
next_page()Optional[SyncPager[T]]Calls get_next() if available and returns the next page object.
__iter__()Iterator[T]Allows iteration through all items across all pages using for loops.
iter_pages()Iterator[SyncPager[T]]Allows iteration through page objects themselves.

The IncompleteUploadSummary class contains the following properties:

NameTypeDescription
upload_idstrThe unique identifier of your upload session.
statusMultipartUploadStatusTypeThe current status of the upload session.
total_sizeintTotal size of the file in bytes.
chunk_sizeintThe size of each chunk in bytes.
total_chunksintThe total number of chunks in this upload session.
created_atOptional[datetime]The date and time when the upload session was created.
expires_atOptional[datetime]The date and time when the upload session expires.

API Reference: List incomplete uploads

Create a multipart upload session

Description: This method creates a multipart upload session.

Supported content: Video and audio

File size: 4GB maximum.

Additional requirements depend on your workflow:

Function signature and example:

1def create(
2 self,
3 *,
4 filename: str,
5 total_size: int,
6 request_options: typing.Optional[RequestOptions] = None
7) -> CreateAssetUploadResponse

Parameters:

NameTypeRequiredDescription
filenamestrYesThe original file name of the asset.
total_sizeintYesThe total size of the file in bytes. The platform uses this value to:
- Calculate the optimal chunk size.
- Determine the total number of chunks required
- Generate the initial set of presigned URLs
request_optionsRequestOptionsNoRequest-specific configuration.

Return value: Returns a CreateAssetUploadResponse object containing details about the upload session.

The CreateAssetUploadResponse class contains the following properties:

NameTypeDescription
upload_idOptional[str]The unique identifier of this upload session. Store this value, as you’ll need it for the following subsequent operations:
- Reporting completed chunks
- Requesting additional presigned URLs
- Retrieving the status of this upload session

This identifier remains valid for 24 hours from the time of creation.
asset_idOptional[str]The unique identifier for the asset being created. Store this value, as you’ll need it to reference the asset in other API calls. Note that this identifier is reserved immediately, but the asset becomes available for other operations only after the upload is completed successfully.
upload_urlsOptional[List[PresignedUrlChunk]]The initial set of presigned URLs for uploading chunks. Each URL corresponds to a specific chunk.

NOTES:
- URLs expire after one hour.
- Depending on the size of the file, this initial set may not include URLs for all chunks. You can request more using the get_additional_presigned_urls method.
upload_headersOptional[Dict[str, str]]Headers to include when uploading chunks to the presigned URLs.
chunk_sizeOptional[int]The size in bytes for each chunk, except for the last chunk, which may be smaller. Use this value to divide your file into chunks of this exact size.
total_chunksOptional[int]The total number of chunks into which your file must be split. Calculated as: ceiling(total_size / chunk_size).
expires_atOptional[datetime]A string representing the date and time, in RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”), when the upload URL will expire. Upload URLs expire 24 hours from their creation. After expiration, you cannot resume the upload, and you must create a new upload session.

The PresignedUrlChunk class contains the following properties:

NameTypeDescription
chunk_indexOptional[int]The index of this chunk.
urlOptional[str]The presigned URL for uploading this chunk. Each URL can only be used once and expires after one hour.
expires_atOptional[datetime]The date and time when the presigned URL expires.

API Reference: Create a multipart upload session

Retrieve the status of an upload session

Description: This method retrieves information about an upload session, including its current status, chunk-level progress, and completion state.

Use this method to:

  • Verify upload completion (status = completed)
  • Identify any failed chunks that require a retry
  • Monitor the upload progress by comparing uploaded_size with total_size
  • Determine if the session has expired
  • Retrieve the status information for each chunk

You must call this method after reporting chunk completion to confirm the upload has transitioned to the completed status before using the asset.

Function signature and example:

1def get_status(
2 self,
3 upload_id: str,
4 *,
5 page: typing.Optional[int] = None,
6 page_limit: typing.Optional[int] = None,
7 request_options: typing.Optional[RequestOptions] = None,
8) -> SyncPager[ChunkInfo]

Parameters:

NameTypeRequiredDescription
upload_idstrYesThe unique identifier of the upload session.
pageintNoA number that identifies the page to retrieve. Default: 1.
page_limitintNoThe number of items to return on each page. Default: 10. Max: 50.
request_optionsRequestOptionsNoRequest-specific configuration.

Return value: Returns a SyncPager[ChunkInfo] object that allows you to iterate through the paginated chunk status information.

The SyncPager[T] class contains the following properties and methods:

NameTypeDescription
itemsOptional[List[T]]A list containing the current page of items. Can be None.
has_nextboolIndicates whether there is a next page to load.
get_nextOptional[Callable[[], Optional[SyncPager[T]]]]A callable function that retrieves the next page. Can be None.
responseOptional[BaseHttpResponse]The HTTP response object. Can be None.
next_page()Optional[SyncPager[T]]Calls get_next() if available and returns the next page object.
__iter__()Iterator[T]Allows iteration through all items across all pages using for loops.
iter_pages()Iterator[SyncPager[T]]Allows iteration through page objects themselves.

The ChunkInfo class contains the following properties:

NameTypeDescription
indexOptional[int]The index of the chunk. The platform uses 1-based indexing, and this value matches the value of the chunk_index field in the list of upload URLs.
statusOptional[ChunkInfoStatus]The current status of this chunk. Values:
- completed: Successfully uploaded and reported.
- pending: Not yet reported. A chunk may be in this status if it has been uploaded but not yet reported.
- failed: The upload process failed; you must retry uploading this chunk.
uploaded_atOptional[datetime]The date and time, in the RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”), when this chunk was successfully reported as uploaded. The value of this field is null for pending or failed chunks.
updated_atOptional[datetime]The date and time when this chunk was last updated.
errorOptional[str]A detailed error message explaining why this chunk failed. The platform returns this field only when the status is failed.

API Reference: Retrieve the status of an upload session

Report uploaded chunks

Description: This method reports successfully uploaded chunks to the platform. The platform finalizes the upload after you report all chunks.

For optimal performance, report chunks in batches and in any order.

Function signature and example:

1def report_chunk_batch(
2 self,
3 upload_id: str,
4 *,
5 completed_chunks: typing.Sequence[CompletedChunk],
6 request_options: typing.Optional[RequestOptions] = None,
7) -> ReportChunkBatchResponse

Parameters:

NameTypeRequiredDescription
upload_idstrYesThe unique identifier of the upload session.
completed_chunksSequence[CompletedChunk]YesThe list of chunks successfully uploaded that you’re reporting to the platform. Report only after receiving an ETag.
request_optionsRequestOptionsNoRequest-specific configuration.

The CompletedChunk class contains the following properties:

NameTypeDescription
chunk_indexintThe number that identifies which chunk you uploaded. When you received presigned URLs from the platform, each URL was assigned an index number. Use that same number. The chunks are numbered starting from 1.
proofstrThe ETag value you received after uploading the chunk. When you upload a chunk to a presigned URLs, the response includes an ETag. Use this value and submit it as proof of successful upload.
proof_typeLiteral["etag"]The verification method. Supported value: etag. Default: etag.
chunk_sizeintThe number of bytes uploaded for this chunk. For all chunks except the last, this value equals chunk_size. For the last chunk, it may be smaller.

Return value: Returns a ReportChunkBatchResponse object containing information about the reported chunks.

The ReportChunkBatchResponse class contains the following properties:

NameTypeDescription
urlOptional[str]The URL for accessing your asset. The platform returns this field only when all chunks are reported and the upload is complete. If absent, continue uploading and reporting the remaining chunks.
asset_idOptional[str]The unique identifier of this asset.
processed_chunksOptional[int]The number of chunks accepted from this specific request. This equals the number of chunks in your completed_chunks array minus any duplicates.
duplicate_chunksOptional[int]The number of chunks in this request that were already reported. Duplicates are ignored and don’t affect your upload.
total_completedOptional[int]The cumulative count of all unique chunks successfully reported across all requests. When this equals total_chunks, the upload is complete.

API Reference: Report uploaded chunks

Request presigned URLs for the remaining chunks

Description: This method generates new presigned URLs for specific chunks that require uploading. Use this method in the following situations:

  • Your initial URLs have expired (URLs expire after one hour).
  • The initial set of presigned URLs does not include URLs for all chunks.
  • You need to retry failed chunk uploads with new URLs.

To specify which chunks need URLs, use the start and count parameters. For example, to generate URLs for chunks 21 to 30, use start=21 and count=10.

The response provides new URLs, each with a fresh expiration time of one hour.

Function signature and example:

1def get_additional_presigned_urls(
2 self,
3 upload_id: str,
4 *,
5 start: int,
6 count: int,
7 request_options: typing.Optional[RequestOptions] = None
8) -> RequestAdditionalPresignedUrLsResponse

Parameters:

NameTypeRequiredDescription
upload_idstrYesThe unique identifier of the upload session.
startintYesThe index of the first chunk number to generate URLs for. Chunks are numbered from 1.
countintYesThe number of presigned URLs to generate starting from the index. You can request a maximum of 50 URLs in a single API call.
request_optionsRequestOptionsNoRequest-specific configuration.

Return value: Returns a RequestAdditionalPresignedUrLsResponse object containing the new presigned URLs.

The RequestAdditionalPresignedUrLsResponse class contains the following properties:

NameTypeDescription
upload_idOptional[str]The unique identifier of the upload session associated with these URLs.
start_indexOptional[int]The index of the first chunk number in this set of URLs. Matches the start value from your request.
countOptional[int]The number of new URLs created. Matches the count value from your request.
upload_urlsOptional[List[PresignedUrlChunk]]An array of additional presigned URLs for uploading chunks.
generated_atOptional[datetime]The date and time, in the RFC 3339 format (“YYYY-MM-DDTHH:mm:ssZ”), when these URLs were created. URLs remain valid for 1 hour from this time.
expires_atOptional[datetime]The date and time when the upload session expires.

API Reference: Request presigned URLs for the remaining chunks