Multipart uploads
The MultipartUploadClient class provides methods to manage multipart upload sessions.
Utilize multipart uploads to upload large files by dividing them into smaller chunks. This approach enables reliable uploads of large files, particularly when processing chunks in parallel or resuming interrupted transfers. TwelveLabs recommends using this method for files larger than 200 MB, with a maximum size limit of 4 GB.
This method creates an asset that you can use in different workflows.
Workflow
Determine the total size of your file in bytes. You’ll need this value when creating the upload session.
Create an upload session: Call the multipart_upload.create method, providing details about your file, including its size. The response contains, among other information, the unique identifier of the asset, a list of upload URLs, and the size of each chunk in bytes.
Split your file: Use the chunk size from the response to divide your file into chunks of the specified size.
Upload chunks: Transfer each chunk to its designated presigned URL. You can upload the chunks in parallel for improved performance. Save the ETag from each upload response for progress reporting.
(Optional) Request additional URLs: Call the multipart_upload.get_additional_presigned_urls if you need URLs for remaining chunks or if existing URLs expire.
Report progress: Submit completed chunks via the multipart_upload.report_chunk_batch method in batches as chunks finish uploading. Use the ETag from each chunk upload as proof of successful transfer.
Confirm completion: The upload is complete when the multipart_upload.get_status mathod returns status: 'completed'. Use the asset ID from step 1 to perform additional operations on your uploaded video.
What you do next depends on your use case:
- For creating embeddings (videos, audio, images): Use the asset ID with the Embed API v2.
- For search and analysis (videos): Index an asset using the asset ID.
Methods
List incomplete uploads
Description: This method returns a list of all incomplete multipart upload sessions in your account.
Function signature and example:
Parameters:
Return value: Returns a SyncPager[IncompleteUploadSummary] object that allows you to iterate through the paginated results.
The SyncPager[T] class contains the following properties and methods:
The IncompleteUploadSummary class contains the following properties:
API Reference: List incomplete uploads
Create a multipart upload session
Description: This method creates a multipart upload session.
Supported content: Video and audio
File size: 4GB maximum.
Additional requirements depend on your workflow:
- Search: Marengo requirements
- Video analysis: Pegasus requirements
- Create embeddings: Marengo requirements
Function signature and example:
Parameters:
Return value: Returns a CreateAssetUploadResponse object containing details about the upload session.
The CreateAssetUploadResponse class contains the following properties:
The PresignedUrlChunk class contains the following properties:
API Reference: Create a multipart upload session
Retrieve the status of an upload session
Description: This method retrieves information about an upload session, including its current status, chunk-level progress, and completion state.
Use this method to:
- Verify upload completion (
status=completed) - Identify any failed chunks that require a retry
- Monitor the upload progress by comparing
uploaded_sizewithtotal_size - Determine if the session has expired
- Retrieve the status information for each chunk
You must call this method after reporting chunk completion to confirm the upload has transitioned to the completed status before using the asset.
Function signature and example:
Parameters:
Return value: Returns a SyncPager[ChunkInfo] object that allows you to iterate through the paginated chunk status information.
The SyncPager[T] class contains the following properties and methods:
The ChunkInfo class contains the following properties:
API Reference: Retrieve the status of an upload session
Report uploaded chunks
Description: This method reports successfully uploaded chunks to the platform. The platform finalizes the upload after you report all chunks.
For optimal performance, report chunks in batches and in any order.
Function signature and example:
Parameters:
The CompletedChunk class contains the following properties:
Return value: Returns a ReportChunkBatchResponse object containing information about the reported chunks.
The ReportChunkBatchResponse class contains the following properties:
API Reference: Report uploaded chunks
Request presigned URLs for the remaining chunks
Description: This method generates new presigned URLs for specific chunks that require uploading. Use this method in the following situations:
- Your initial URLs have expired (URLs expire after one hour).
- The initial set of presigned URLs does not include URLs for all chunks.
- You need to retry failed chunk uploads with new URLs.
To specify which chunks need URLs, use the start and count parameters. For example, to generate URLs for chunks 21 to 30, use start=21 and count=10.
The response provides new URLs, each with a fresh expiration time of one hour.
Function signature and example:
Parameters:
Return value: Returns a RequestAdditionalPresignedUrLsResponse object containing the new presigned URLs.
The RequestAdditionalPresignedUrLsResponse class contains the following properties:
API Reference: Request presigned URLs for the remaining chunks