TwelveLabs’ video understanding models consist of a family of deep neural networks built on our multimodal foundation model for video understanding that you can use for the following downstream tasks:

Search using natural language queries
Generate text from video.

Videos contain multiple types of information, including visuals, sounds, spoken words, and texts. The human brain combines all types of information and their relations with each other to comprehend the overall meaning of a scene. For example, you’re watching a video of a person jumping and clapping, both visual cues, but the sound is muted. You might realize they’re happy, but you can’t understand why they’re happy without the sound. However, if the sound is unmuted, you could realize they’re cheering for a soccer team that scored a goal.

Thus, an application that analyzes a single type of information can’t provide a comprehensive understanding of a video. TwelveLabs’ video understanding models, however, analyze and combine information from all the modalities to accurately interpret the meaning of a video holistically, similar to how humans watch, listen, and read simultaneously to understand videos.

Our video understanding models have the ability to identify, analyze, and interpret a variety of elements, including but not limited to the following:

Element	Modality	Example
People, including famous individuals	Visual	Michael Jordan, Steve Jobs
Actions	Visual	Running, dancing, kickboxing
Objects	Visual	Cars, computers, stadiums
Animals or pets	Visual	Monkeys, cats, horses
Nature	Visual	Mountains, lakes, forests
Text displayed on the screen (OCR)	Visual	License plates, handwritten words, number on a player’s jersey
Brand logos	Visual	Nike, Starbucks, Mercedes
Shot techniques and effects	Visual	Aerial shots, slow motion, time-lapse
Counting objects	Visual	Number of people in a crowd, items on a shelf, vehicles in traffic
Sounds	Audio	Chirping (birds), applause, fireworks popping or exploding
Human speech	Audio	”Good morning. How may I help you?”
Music	Audio	Ominous music, whistling, lyrics

Supported languages

The platform supports the following languages for processing visual and audio content, understanding queries or prompts, and generating outputs:

Full support: English
Partial support: Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Vietnamese