Cloud-to-cloud integrations

Note

Cloud-to-cloud integrations require a paid plan. If you’re on the Free plan, you can find information on upgrading your plan in the Upgrade your plan section.

Cloud-to-cloud integrations allow you to upload multiple videos in a single API call. This feature is currently supported for the us-west-2 region of AWS S3. If your data is in other regions or with other cloud providers, contact us at sales@twelvelabs.io.

Note the following about cloud-to-cloud integrations:

  • The import process is asynchronous. Your videos will be uploaded and indexed after you initiate an import.
  • You can perform downstream tasks on your videos only after the platform has finished uploading and indexing them.
  • Only one import job can run at a time. To start a new import, wait for the current job to complete. Use the GET method of the /tasks/transfers/import/{integration-id}/logs endpoint to retrieve a list of your import jobs, including their creation time, completion time, and processing status for each video file.

The steps for importing videos are as follows:

Set up an integration
Import videos

Set up an integration

Setting up an integration grants TwelveLabs access to read files in your S3 bucket. The platform assigns a unique identifier to each integration. Before you can use this feature, you must set up an integration by following the steps in this section.

Prerequisites

Ensure the following prerequisites are met:

  • Your bucket must be in the us-west-2 region.
  • You have the following information on hand:
    • Your AWS account ID. For instructions, see the Finding Your AWS Account ID section of the AWS documentation.
    • The name of your bucket.

Procedure

1

Go to the Integrations page.

2

Select Add Integration > AWS S3:

3

In the Integrate AWS S3 modal, enter your AWS ID and bucket name. Then, select the Next button:

Create an AWS policy

1

Open the IAM Dashboard page in a new window or tab.

2

From the sidebar, choose Policies. Then, select the Create Policy button:

3

Select the JSON tab:

4

Move to the TwelveLabs Dashboard page, and copy the JSON snippet under Step 1 : Create Policy:

5

Move to the IAM Dashboard page, and replace the content of the JSON tab with the snippet you’ve copied in the previous step. Then, select the Next button:

6

On the Review Policy page, set the name of the policy to “TwelvelabsIntegrationPolicy” and enter a brief description. Then, select the Create Policy button:

Note

Do not change the name of the policy. If you rename the policy, the integration will not work.

7

On the Policies page, make sure that the system displays your new policy: If the system does not display your policy, review the steps in this section, making sure all the information you entered is correct.

Create an AWS role

1

From the sidebar, choose Roles. Then, select the Create Role button:

2

Under Trusted Entity Type, select Custom Trust Policy:

3

Move to the TwelveLabs Integrations page, and copy the JSON snippet under Step 2 : Create Role:

4
  1. Move to the IAM Dashboard page, and replace the content of the Custom trust policy box with the snippet you’ve copied in the previous step. Then, select the Next button:
5

On the Add Permissions page, select the policy you’ve created in the previous section. Then, select the Next button at the bottom-right corner of the page:

6

On the Name, Review, and Create page, set the name of the role to “TwelvelabsIntegrationRole” and enter a brief description. Then, select the Create Role button:

Note

Do not change the name of the role. If you rename the role, the integration will not work.

7

On the Roles page, make sure that the system displays your new role:

If the system does not display your role, review the steps in this section, making sure all the information you entered is correct.

Verify your integration configuration

1

Move to the TwelveLabs Integrations page. Under Step 3 : Check Verification, select the Verify My Account button:

If everything went well, you should see a message saying “Verification successful!”:

2

Select the Next button.

3

On the Integrations page, make sure that the status of your new integration shows as Active. Note that each integration has a unique identifier that you must provide to import videos.

When the status of your integration shows as Active, the platform can access the videos in your S3 bucket. If the status does not show as Active, review the steps in this section, making sure all the information you entered is correct.

Import videos

You can import videos in two ways: 

  • Programmatically: Follow the steps in this guide.
  • Using the Playground: Go to the Integrations page, choose your integration, and select the Import Videos button.

Prerequisites

Before you import videos, ensure the following prerequisites are met:

  • You’ve already created and index.
  • You have an AWS S3 bucket containing the videos you want to import.
  • You’ve already set up an integration. If you didn’t set up an integration yet, follow the steps in the Set up an integration section.
  • Retrieve the unique identifier of your integration from the Integrations page:
  • Your videos must meet the requirements in the Prerequisites section of the Upload single videos page.

Procedure

1

To initiate an import, invoke the import_video method, passing the unique identifier of your integration and the unique identifier of the target index as parameters:

1from twelvelabs import TwelveLabs
2
3client = TwelveLabs(api_key="<YOUR_API_KEY>")
4
5res = client.task.transfers.import_videos(
6"<YOUR_INTEGRATION_ID>",
7"<YOUR_INDEX_ID>",
8)
9for video in res.videos:
10print(f"video: {video.video_id} {video.filename}")
11if res.failed_files:
12for failed_file in res.failed_files:
13    print(f"failed_file: {failed_file.filename} {failed_file.error_message}")

Note the following about the response:

  • It consists of two lists:
    • videos: Videos that will be imported.
    • failed_files: Videos that failed to import, typically due to unmet prerequisites .
  • The platform returns this information immediately after initiating the import process, before the actual upload and indexing begin. For details about each field in the request and response, see the Import videos page.
2

(Optional) Monitor the status of your import by invoking the import_status method, passing the unique identifier of your integration and the unique identifier of the target index as parameters:

1status = client.task.transfers.import_status("<YOUR_INTEGRATION_ID>", "<YOUR_INDEX_ID>")
2for ready in status.ready:
3print(f"ready: {ready.video_id} {ready.filename} {ready.created_at}")
4for failed in status.failed:
5print(f"failed: {failed.filename} {failed.error_message}")

Note the following about the response:

  • It consists of lists of videos grouped by their status. For details about each status, see the Task object page.
  • Each video entry includes:
    • video_id: The unique identifier of the video. This identifier serves a dual purpose:
    • It identifies the video itself.
    • It identifies the associated video indexing task.
    • filename: The original filename of the video.
    • created_at: The date and time when the video was added to the import process.

For details about each field in the request and response, see the Retrieve import status page.

Duplicate videos

By default, the platform checks for duplicate files using hashes within the target index and will not upload the same video to the same index twice. However, the same video can exist in multiple indexes.

To bypass duplicate checking entirely and import duplicate videos into the same index, set the value of the incremental_import parameter to false, as shown in the example code below:

1res = client.task.transfers.import_videos(
2   integration_id="<YOUR_INTEGRATION_ID>",
3   index_id="<YOUR_INDEX_ID>",
4   incremental_import=False  # Allows duplicate videos in the same index
5)

Troubleshooting

To view a history of the import operations for a specific integration, invoke the import_log method, passing the unique identifier of your integration as a parameter:

1logs = client.task.transfers.import_logs("<YOUR_INTEGRATION_ID>")
2for log in logs:
3    print(
4        f"index_id={log.index_id} index_name={log.index_name} created_at={log.created_at} ended_at={log.ended_at} video_status={log.video_status}"
5    )
6    if log.failed_files:
7        for failed_file in log.failed_files:
8            print(
9                f"failed_file: {failed_file.filename} {failed_file.error_message}"
10            )

The response consists of a chronological list of import operations for the specified integration. The list is sorted by creation date, with the oldest imports first. Each item in the list contains:

  • The number of videos in each status
  • Detailed error information for failed uploads, including filenames and error messages.

For details about each field in the request and response, see the Retrieve import logs page.

Built with