Create audio embeddings
This guide shows how you can create audio embeddings.
The following table lists the available models for generating video embeddings and their key characteristics:
Model | Description | Dimensions | Max length | Similarity metric |
---|---|---|---|---|
Marengo-retrieval-2.6 | Use this model to create embeddings that you can use in various downstream tasks | 1024 | 10 seconds | Cosine similarity |
Note that the “Marengo-Retrieval-2.6” video understanding engine generates embeddings for all modalities in the same latent space. This shared space enables any-to-any searches across different types of content.
To create an embedding for an audio file, you must specify at least the following parameters:
engine_name
: The name of the video understanding engine you want to use. Example: "Marengo-retrieval-2.6".audio_url
oraudio_file
: The publicly accessible URL of your audio file or the path of your audio file. You must provide at least one of these parameters. If you specify both, theaudio_url
parameter takes precedence.
The response is an object containing the following fields:
engine_name
: The name of the engine the platform has used to create this embedding.audio_embedding
: An object that contains your embeddings and additional information.
For a description of each field in the request and response, see the Create embeddings for text, image, and audio page.
Prerequisites
- You’re familiar with the concepts that are described on the Platform overview page.
- You have an API key. To retrieve your API key, navigate to the API Key page and log in with your credentials. Then, select the Copy icon to the right of your API key to copy it to your clipboard.
- The audio files you wish to use must meet the following requirements:
- Format: WAV (uncompressed), MP3 (lossy), and FLAC (lossless)
- Duration: Must not exceed 10 seconds.
Example
The example code below creates an audio embedding using the default behavior for handling audio files that are too long. Ensure you replace the placeholders surrounded by <>
with your values:
from twelvelabs import TwelveLabs
embedding = client.embed.create(
engine_name="Marengo-retrieval-2.6",
audio_file="<YOUR_AUDIO_FILE>",
)
print("Created an audio embedding")
print(f" Engine: {embedding.engine_name}")
if embedding.audio_embedding.segments:
print("Segments:")
for i, segment in enumerate(embedding.audio_embedding.segments, 1):
print(f" Segment {i}:")
print(f" Start Offset (sec): {segment.start_offset_sec}")
if segment.values:
print(f" Values: {segment.values[:5]}... (truncated)")
import { TwelveLabs } from 'twelvelabs';
const client = new TwelveLabs({ apiKey:'<YOUR_API_KEY>' });
const embedding = await client.embed.create({
engineName: 'Marengo-retrieval-2.6',
audioFile: '<YOUR_AUDIO_FILE>',
});
console.log("Created an audio embedding");
console.log(` Engine: ${embedding.engineName}`);
console.log(` Embedding: ${embedding.audioEmbedding.float}`);
Updated 12 days ago