This guide shows you how to use TwelveLabs models on Amazon Bedrock to perform the following tasks:

• Analyze videos and generate text based on their content.
• Create embeddings from video, text, audio, or image inputs.

TwelveLabs provides the following models on Amazon Bedrock:

The table below shows the specifications for each type of embedding:

For more information, see the TwelveLabs in Amazon Bedrock product page. For details on pricing, see the Amazon Bedrock pricing page.

Prerequisites

Before you start, ensure you have the following:

• An AWS account with access to a region where the TwelveLabs models are supported.
• An AWS IAM principal with sufficient Amazon Bedrock permissions. For details on setting permissions, see the Identity and access management for Amazon Bedrock page.
• S3 permissions to read input files and write output files for Marengo operations.
• The AWS CLI and configured with your credentials.
• Python 3.7 or later with the boto3 library.
• Access to the model you want to use. Navigate to the AWS Console > Bedrock > Model Access page and request access. Note that the availability of the models varies by region.

Media input methods

Both TwelveLabs models support base64 encoded strings and S3 URIs for media input. Note that the base64 method has a 36MB file size limit. This guide uses S3 URIs.

S3 output requirements for Marengo

Marengo uses the StartAsyncInvoke API and requires an S3 location for storing results. Each invocation creates a unique directory with two files:

• manifest.json: Contains metadata including the request ID.
• output.json: Contains the actual embeddings.

Examples

The sections below provide examples of using the TwelveLabs models on Amazon Bedrock.

Analyze videos

The example code below shows how you can use Pegasus to analyze videos and generate text based on their content. Ensure you replace the following placeholders with your values:

• <YOUR_REGION>: with your region (example: “eu-west-1”).
• <YOUR_ACCOUNT_ID>: with your AWS account ID (example: “123456789012”).
• <YOUR_BUCKET_NAME>: with the name of your S3 bucket (example: “my-video-bucket”).
• <YOUR_VIDEO_FILE>: with the name of your video file (example: “my_video.mp4”).
• <YOUR_PROMPT>: with a string that guides the model on the desired format or content (example: “Summarize this video”).

1
import boto3
2
import json
3
import base64
4
5
REGION = "<YOUR_REGION>"
6
MODEL_ID = "eu.twelvelabs.pegasus-1-2-v1:0"
7
INPUT_PROMPT = "<YOUR_PROMPT>"
8
ACCOUNT_ID = "<YOUR_ACCOUNT_ID>"
9
BUCKET = "<YOUR_BUCKET_NAME>"
10
FILE_NAME = "<YOUR_VIDEO_FILE>"
11
12
bedrock = boto3.client(service_name="bedrock-runtime", region_name=REGION)
13
body = json.dumps({
14
    "inputPrompt": INPUT_PROMPT,
15
    "mediaSource": {
16
        "s3Location": {
17
            "uri": f"s3://{BUCKET}/{FILE_NAME}",
18
            "bucketOwner": ACCOUNT_ID
19
        }
20
    }
21
})
22
23
response = bedrock.invoke_model(
24
    body=body,
25
    modelId=MODEL_ID,
26
    contentType="application/json",
27
    accept="application/json"
28
)
29
30
response_body = json.loads(response.get("body").read())
31
print(response_body)

Create embeddings from video, audio, or image inputs

The example code below shows how you can use Marengo to create embeddings from video, audio, or image inputs. Marengo uses the StartAsyncInvoke API, which means you must:

1. Submit your request, providing an S3 location for your input media file and an S3 location for the output. Note that this example uses the same bucket.
2. Wait for the job to complete.
3. Retrieve the results from the S3 output location.

Ensure you replace the following placeholders with your values:

• <YOUR_REGION>: with your AWS region (example: “eu-west-1”)
• <YOUR_ACCOUNT_ID>: with your AWS account ID (example: “123456789012”)
• <YOUR_BUCKET_NAME>: with the name of your S3 bucket (example: “my-bucket”)
• <YOUR_VIDEO_FILE>: with the name of your video file (example: “my_file.mp4”)
• <YOUR_INPUT_TYPE>: with the type of media you wish to provide. The following values are supported: “video”, “audio”, or “image”.

1
import boto3
2
import time
3
4
REGION = "<YOUR_REGION>"
5
MODEL_ID = "twelvelabs.marengo-embed-2-7-v1:0"
6
ACCOUNT_ID = "<YOUR_ACCOUNT_ID>"
7
BUCKET = "<YOUR_BUCKET_NAME>"
8
FILE_NAME = "<YOUR_VIDEO_FILE>"
9
INPUT_TYPE = "<YOUR_INPUT_TYPE>"
10
11
bedrock_client = boto3.client(service_name="bedrock-runtime", region_name=REGION)
12
13
# Start async video embedding
14
model_input = {
15
    "mediaSource": {
16
        "s3Location": {
17
            "uri": f"s3://{BUCKET}/{FILE_NAME}",
18
            "bucketOwner": ACCOUNT_ID
19
        }
20
    },
21
    "inputType": INPUT_TYPE
22
}
23
24
async_request_response = bedrock_client.start_async_invoke(
25
    modelId=MODEL_ID,
26
    modelInput=model_input,
27
    outputDataConfig={
28
        "s3OutputDataConfig": {
29
            "s3Uri": f"s3://{BUCKET}",
30
            "bucketOwner": ACCOUNT_ID
31
        }
32
    }
33
)
34
35
print("async_request_response: ", async_request_response)
36
37
# get the invocation arn
38
invocation_arn = async_request_response.get("invocationArn")
39
40
# wait for the async job to complete
41
max_retries = 60
42
retries = 0
43
while True:
44
    response = bedrock_client.get_async_invoke(
45
        invocationArn=invocation_arn
46
    )
47
    print(f"status: {response.get('status')}")
48
    if response.get("status") == "Completed":
49
        break
50
    time.sleep(1)
51
    retries += 1
52
    if retries > max_retries:
53
        break
54
55
print(response)
56
57
# Extract the S3 URI where results are stored
58
output_s3_uri = response.get("outputDataConfig", {}).get("s3OutputDataConfig", {}).get("s3Uri")
59
print(f"Results stored at: {output_s3_uri}")

Create embeddings from text inputs

The example code below shows how you can use Marengo to create embeddings from text inputs. Marengo uses the StartAsyncInvoke API, which means you must:

1. Submit your request, providing an S3 location for the output.
2. Wait for the job to complete.
3. Retrieve the results from the S3 output location.

Ensure you replace the following placeholders with your values:

• <YOUR_REGION>: with your AWS region (example: “eu-west-1”).
• <YOUR_ACCOUNT_ID>: with your AWS account ID (example: “123456789012”).
• <YOUR_BUCKET_NAME>: with your S3 bucket name (example: “my-bucket”).
• <YOUR_TEXT>: with the text for which you wish to create an embedding (example: “A man walking down the street”).

1
import boto3
2
import time
3
4
REGION = "<YOUR_REGION>"
5
MODEL_ID = "twelvelabs.marengo-embed-2-7-v1:0"
6
ACCOUNT_ID = "<YOUR_ACCOUNT_ID>"
7
BUCKET = "<YOUR_BUCKET_NAME>"
8
9
bedrock_client = boto3.client(service_name="bedrock-runtime", region_name=REGION)
10
11
# Start async video embedding
12
model_input = {
13
    "inputType": "text",
14
    "inputText": "<YOUR_TEXT>"
15
}
16
17
async_request_response = bedrock_client.start_async_invoke(
18
    modelId=MODEL_ID,
19
    modelInput=model_input,
20
    outputDataConfig={
21
        "s3OutputDataConfig": {
22
            "s3Uri": f"s3://{BUCKET}",
23
            "bucketOwner": ACCOUNT_ID
24
        }
25
    }
26
)
27
28
print("async_request_response: ", async_request_response)
29
30
# get the invocation arn
31
invocation_arn = async_request_response.get("invocationArn")
32
33
# wait for the async job to complete
34
max_retries = 60
35
retries = 0
36
while True:
37
    response = bedrock_client.get_async_invoke(
38
        invocationArn=invocation_arn
39
    )
40
    print(f"status: {response.get('status')}")
41
    if response.get("status") == "Completed":
42
        break
43
    time.sleep(1)
44
    retries += 1
45
    if retries > max_retries:
46
        break
47
48
print(response)
49
50
# Extract the S3 URI where results are stored
51
output_s3_uri = response.get("outputDataConfig", {}).get("s3OutputDataConfig", {}).get("s3Uri")
52
print(f"Results stored at: {output_s3_uri}")

Request parameters and response fields

For a complete list of request parameters and response fields, see the following pages in the Amazon Bedrock documentation:

• TwelveLabs Marengo Embed 2.7.
• TwelveLabs Pegasus 1.2.