Qdrant - Building a semantic video search workflow

Summary: This integration combines TwelveLabs’ Embed API with Qdrant Vector Search to create an efficient semantic video search solution. It generates multimodal embeddings for video content, allowing for precise and relevant search results across various modalities.

Description: The process of performing semantic video searches using TwelveLabs and Qdrant involves the following main steps:

  1. Generate multimodal embeddings for your video content.
  2. Store these embeddings in Qdrant.
  3. Create embeddings for your search queries. You can provide either text, audio, or images as queries.
  4. Use these query embeddings to perform vector searches in Qdrant.

Step-by-step guide: Our blog post, Building a Semantic Video Search Workflow with TwelveLabs and Qdrant, guides you through the process of building a semantic video search workflow. This workflow is ideal for applications like video indexing, content recommendation systems, and contextual search engines.

Colab Notebook: TwelveLabs-EmbedAPI-Qdrant

Integration with TwelveLabs

This section explains how to utilize the TwelveLabs Python SDK to create embeddings for semantic video search. The integration involves generating the following types of embeddings:

  • Video embeddings derived from your video content
  • Text, video, and audio embeddings for the queries

Video embeddings

The following code generates an embedding for a video. It creates a video embedding task that processes the video and periodically checks the task’s status to retrieve the embeddings upon completion.

Python
1# Step 1: Create an embedding task
2task = twelvelabs_client.embed.task.create(
3    model_name="Marengo-retrieval-2.7",  # Specify the model
4    video_url="https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_2mb.mp4"  # Video URL
5)
6
7# Step 2: Wait for the task to complete
8task.wait_for_done(sleep_interval=3)  # Check every 3 seconds
9
10# Step 3: Retrieve the embeddings
11task_result = twelvelabs_client.embed.task.retrieve(task.id)
12
13# Display the embedding results
14print("Embedding Vector (First 10 Dimensions):", task_result.embeddings[:10])
15print("Embedding Dimensionality:", len(task_result.embeddings))

For details on creating text embeddings, see the Create video embeddings page.

Text embeddings

The code below generates a text embedding and identifies the video segments that match your text semantically.

Python
1# Generate text embedding
2text_segment = twelvelabs_client.embed.create(
3    model_name="Marengo-retrieval-2.7",
4    text="A white rabbit",  # Input query
5).text_embedding.segments[0]
6
7# Perform semantic search in Qdrant
8text_results = qdrant_client.query_points(
9    collection_name=collection_name,
10    query=text_segment.embeddings_float,  # Use the embedding vector
11)
12
13print("Text Query Results:", text_results)

For details on creating text embeddings, see the Create text embeddings page.

Audio embeddings

The code below generates an audio embedding and finds the video segments that match the semantic content of your audio clip.

Python
1# Generate audio embedding
2audio_segment = twelvelabs_client.embed.create(
3    model_name="Marengo-retrieval-2.7",
4    audio_url="https://codeskulptor-demos.commondatastorage.googleapis.com/descent/background%20music.mp3",  # Audio file URL
5).audio_embedding.segments[0]
6
7# Perform semantic search in Qdrant
8audio_results = qdrant_client.query_points(
9    collection_name=collection_name,
10    query=audio_segment.embeddings_float,  # Use the embedding vector
11)
12
13print("Audio Query Results:", audio_results)

For details on creating text embeddings, see the Create audio embeddings page.

Image embeddings

The code below generates an image embedding and identifies video segments that are semantically similar to the image.

Python
1# Generate image embedding
2image_segment = twelvelabs_client.embed.create(
3    model_name="Marengo-retrieval-2.7",
4    image_url="https://gratisography.com/wp-content/uploads/2024/01/gratisography-cyber-kitty-1170x780.jpg",  # Image URL
5).image_embedding.segments[0]
6
7# Perform semantic search in Qdrant
8image_results = qdrant_client.query_points(
9    collection_name=collection_name,
10    query=image_segment.embeddings_float,  # Use the embedding vector
11)
12
13print("Image Query Results:", image_results)

For details on creating image embeddings, see the Create image embeddings page.

Next steps

After reading this page, you have the following options:

  • Customize and use the example: Use the TwelveLabs-EmbedAPI-Qdrant notebook to understand how the integration works. You can make changes and add functionalities to suit your specific use case.
  • Explore further: Try the applications built by the community or our sample applications to get more insights into the TwelveLabs Video Understanding Platform’s diverse capabilities and learn more about integrating the platform into your applications.
Built with