Summary: This integration combines Twelve Labs' Embed API with ApertureDB to create an efficient semantic video search solution. It captures rich video content as multimodal embeddings, enabling precise and relevant search results.

Description: The process of performing a semantic video search using Twelve Labs and ApertureDB involves two main steps:

Create embeddings for your video content and query
Use the embeddings to perform a vector search in ApertureDB

Step-by-step guide: Our blog post, Semantic Video Search Engine with Twelve Labs and ApertureDB, guides you through the process of creating a video search application, from setup to performing vector searches.

Colab Notebook: TwelveLabs-EmbedAPI-ApertureDB

Integration with Twelve Labs

This section describes how you can use the Twelve Labs Python SDK to create embeddings for semantic video search. The integration involves creating two types of embeddings:

Video embeddings from your video content
Text embeddings from queries

These embeddings form the basis for vector search operations.

Video embeddings

The code below creates a video embedding task that handles the processing of a video. It periodically checks the status of the task and retrieves the embeddings upon completion:

from twelvelabs import TwelveLabs
from twelvelabs.models.embed import EmbeddingsTask

# Initialize the Twelve Labs client
twelvelabs_client = TwelveLabs(api_key=TL_API_KEY)

def generate_embedding(video_url):
    # Create an embedding task
    task = twelvelabs_client.embed.task.create(
        engine_name="Marengo-retrieval-2.6",
        video_url=video_url
    )
    print(f"Created task: id={task.id} engine_name={task.engine_name} status={task.status}")

    # Define a callback function to monitor task progress
    def on_task_update(task: EmbeddingsTask):
        print(f"  Status={task.status}")

    # Wait for the task to complete
    status = task.wait_for_done(
        sleep_interval=2,
        callback=on_task_update
    )
    print(f"Embedding done: {status}")

    # Retrieve the task result
    task_result = twelvelabs_client.embed.task.retrieve(task.id)

    # Extract and return the embeddings
    embeddings = []
    for v in task_result.video_embeddings:
        embeddings.append({
            'embedding': v.embedding.float,
            'start_offset_sec': v.start_offset_sec,
            'end_offset_sec': v.end_offset_sec,
            'embedding_scope': v.embedding_scope
        })

    return embeddings, task_result

# Example usage
video_url = "https://storage.googleapis.com/ad-demos-datasets/videos/Ecommerce%20v2.5.mp4"

# Generate embeddings for the video
embeddings, task_result = generate_embedding(video_url)

print(f"Generated {len(embeddings)} embeddings for the video")
for i, emb in enumerate(embeddings):
    print(f"Embedding {i+1}:")
    print(f"  Scope: {emb['embedding_scope']}")
    print(f"  Time range: {emb['start_offset_sec']} - {emb['end_offset_sec']} seconds")
    print(f"  Embedding vector (first 5 values): {emb['embedding'][:5]}")
    print()

For details on creating video embeddings, see the Create video embeddings page.

Text embeddings

The code below creates a text embedding for the query provided in the text parameter:

# Generate a text embedding for our search query
text_embedding = twelvelabs_client.embed.create(
  engine_name="Marengo-retrieval-2.6",
  text="Show me the part which has lot of outfits being displayed",
  text_truncate="none"
)

print("Created a text embedding")
print(f" Engine: {text_embedding.engine_name}")
print(f" Embedding: {text_embedding.text_embedding.float[:5]}...")  # Display first 5 values

For details on creating text emebddings, see the Create text embeddings page.

Next steps

After reading this page, you have the following options:

Customize and use the example: Use the TwelveLabs-EmbedAPI-ApertureDB notebook to understand how the integration works. You can make changes and add functionalities to suit your specific use case. Below are a few examples:
- Explore data modeling: Experiment with different video segmentation strategies to optimize embedding generation.
- Implement advanced search: Try multimodal queries combining text, image, and audio inputs.
- Scale your system: Test performance with larger video datasets and optimize for high-volume queries.
Explore further: Try the applications built by the community or our sample applications to get more insights into the Twelve Labs Video Understanding Platform's diverse capabilities and learn more about integrating the platform into your applications.