ApertureDB - Semantic video search engine

Summary: This integration combines TwelveLabs’ Embed API with ApertureDB to create an efficient semantic video search solution. It captures rich video content as multimodal embeddings, enabling precise and relevant search results.

Description: The process of performing a semantic video search using TwelveLabs and ApertureDB involves two main steps:

  1. Create embeddings for your video content and query
  2. Use the embeddings to perform a vector search in ApertureDB

Step-by-step guide: Our blog post, Semantic Video Search Engine with TwelveLabs and ApertureDB, guides you through the process of creating a video search application, from setup to performing vector searches.

Colab Notebook: TwelveLabs-EmbedAPI-ApertureDB

Integration with TwelveLabs

This section describes how you can use the TwelveLabs Python SDK to create embeddings for semantic video search. The integration involves creating two types of embeddings:

  • Video embeddings from your video content
  • Text embeddings from queries

These embeddings form the basis for vector search operations.

Video embeddings

The code below creates a video embedding task that handles the processing of a video. It periodically checks the status of the task and retrieves the embeddings upon completion:

Python
1from twelvelabs import TwelveLabs
2from twelvelabs.models.embed import EmbeddingsTask
3
4# Initialize the TwelveLabs client
5twelvelabs_client = TwelveLabs(api_key=TL_API_KEY)
6
7def generate_embedding(video_url):
8    # Create an embedding task
9    task = twelvelabs_client.embed.task.create(
10        engine_name="Marengo-retrieval-2.6",
11        video_url=video_url
12    )
13    print(f"Created task: id={task.id} engine_name={task.engine_name} status={task.status}")
14
15    # Define a callback function to monitor task progress
16    def on_task_update(task: EmbeddingsTask):
17        print(f"  Status={task.status}")
18
19    # Wait for the task to complete
20    status = task.wait_for_done(
21        sleep_interval=2,
22        callback=on_task_update
23    )
24    print(f"Embedding done: {status}")
25
26    # Retrieve the task result
27    task_result = twelvelabs_client.embed.task.retrieve(task.id)
28
29    # Extract and return the embeddings
30    embeddings = []
31    for v in task_result.video_embeddings:
32        embeddings.append({
33            'embedding': v.embedding.float,
34            'start_offset_sec': v.start_offset_sec,
35            'end_offset_sec': v.end_offset_sec,
36            'embedding_scope': v.embedding_scope
37        })
38
39    return embeddings, task_result
40
41# Example usage
42video_url = "https://storage.googleapis.com/ad-demos-datasets/videos/Ecommerce%20v2.5.mp4"
43
44# Generate embeddings for the video
45embeddings, task_result = generate_embedding(video_url)
46
47print(f"Generated {len(embeddings)} embeddings for the video")
48for i, emb in enumerate(embeddings):
49    print(f"Embedding {i+1}:")
50    print(f"  Scope: {emb['embedding_scope']}")
51    print(f"  Time range: {emb['start_offset_sec']} - {emb['end_offset_sec']} seconds")
52    print(f"  Embedding vector (first 5 values): {emb['embedding'][:5]}")
53    print()

For details on creating text emebddings, see the Create video embeddings page.

Text embeddings

The code below creates a text embedding for the query provided in the text parameter:

Python
1# Generate a text embedding for our search query
2text_embedding = twelvelabs_client.embed.create(
3  engine_name="Marengo-retrieval-2.6",
4  text="Show me the part which has lot of outfits being displayed",
5  text_truncate="none"
6)
7
8print("Created a text embedding")
9print(f" Engine: {text_embedding.engine_name}")
10print(f" Embedding: {text_embedding.text_embedding.float[:5]}...")  # Display first 5 values

For details on creating text embeddings, see the Create text embeddings page.

Next steps

After reading this page, you have the following options:

  • Customize and use the example: Use the TwelveLabs-EmbedAPI-ApertureDB notebook to understand how the integration works. You can make changes and add functionalities to suit your specific use case. Below are a few examples:
    • Explore data modeling: Experiment with different video segmentation strategies to optimize embedding generation.
    • Implement advanced search: Try multimodal queries combining text, image, and audio inputs.
    • Scale your system: Test performance with larger video datasets and optimize for high-volume queries.
  • Explore further: Try the applications built by the community or our sample applications to get more insights into the TwelveLabs Video Understanding Platform’s diverse capabilities and learn more about integrating the platform into your applications.
Built with