Summary: This integration allows you to create advanced video understanding and retrieval applications. It combines two key components:

• Twelve Labs' Embed API: Generates multimodal embeddings for video content and text.
• LanceDB: A serverless vector database that stores, indexes, and queries high-dimensional vectors at scale.

Key use cases include:

• Semantic video search engines
• Content-based recommendation systems
• Anomaly detection in video streams.

Description: The process of performing a semantic video search using Twleve Labs and LanceDB involves two main steps:

1. Use the Embed API to create multimodal embeddings for video content and text queries.
2. Use the embeddings to perform similarity searches in LanceDB.

Step-by-step guide: Our blog post, Building Advanced Video Understanding ApplicationsL Integrating Twelve Labs Embed API with LanceDB for Multimodal AI, guides you through the process of creating a video search application, from setup to generating video embeddings and querying them efficiently.
Colab Notebook: TwelveLabs-EmbedAPI-LanceDB

Integration with TwelveLabs

This section describes how you can use the Twelve Labs Python SDK to create embeddings for videos and text queries.

Video embeddings

The generate_embedding function takes the URL of a video as a parameter and returns a list of dictionaries, each containing an embedding vector and the associated metadata:

from twelvelabs.models.embed import EmbeddingsTask

def generate_embedding(video_url: str) -> tuple[List[Dict[str, Any]], Any]:
    """Generate embeddings for a given video URL."""
    task = twelvelabs_client.embed.task.create(
        engine_name="Marengo-retrieval-2.6",
        video_url=video_url
    )
    
    def on_task_update(task: EmbeddingsTask):
        print(f"  Status={task.status}")

    task.wait_for_done(sleep_interval=2, callback=on_task_update)
    task_result = twelvelabs_client.embed.task.retrieve(task.id)

    embeddings = [{
        'embedding': v.embedding.float,
        'start_offset_sec': v.start_offset_sec,
        'end_offset_sec': v.end_offset_sec,
        'embedding_scope': v.embedding_scope
    } for v in task_result.video_embeddings]
    
    return embeddings, task_result

For more details, see the Create video embeddings page.

Text embeddings

The get_text_embedding function generates embeddings for text queries:

def get_text_embedding(text_query: str) -> List[float]:
    """Generate a text embedding for a given text query."""
    return twelvelabs_client.embed.text(text_query).embedding.float

For more details, see the Create text embeddings page.

Next steps

After reading this page, you have the following options:

• Customize and use the example: Use the TwelveLabs-EmbedAPI-LanceDB notebook to understand how the integration works. You can make changes and add more functionalities to suit your specific use case. Some notable examples include:
  • Explore advanced features in LanceDB like hybrid search combining vector and metadata filtering.
  • Implement a continuous user feedback loop to improve your search and recommendation results.
• Explore further: Try the applications built by the community or our sample applications to get more insights into the Twelve Labs Video Understanding Platform's diverse capabilities and learn more about integrating the platform into your applications.