The applications on this page demonstrate how the TwelveLabs Video Understanding Platform enhances sports analysis and viewer engagement.

Chessboxing AI Clips

Summary: “Chessboxing AI Clips” is a web application that enables you to semantically search, preview, and compile video clips from the hybrid sport of chessboxing, showcasing the adaptability of the TwelveLabs Video Understanding Platform to unconventional sports.

Description: The application, developed during a TwelveLabs Hackathon, is a React and TypeScript-based web application that offers the following key features:

Semantic video search: You can search for specific moments in chessboxing videos using natural language queries.
Clip preview: Before selecting clips for compilation, you can preview them using the integrated video player.
Highlight reel creation: You can select multiple clips to create custom highlight reels and add AI-generated background music.
Shareable content: The application generates unique URLs for each highlight reel, allowing you to share your compilations on social media platforms.

The application was developed by Ray Deck and Akshay Rakheja.

GitHub: Chessboxing AI Clips
Demo: Chessboxing AI Clips

Integration with TwelveLabs

The code below iterates through a list of video files, uploading each to the TwelveLabs Video Understanding platform if it hasn’t been uploaded yet. It invokes the client.task.create function of the TwelveLabs Python SDK to create a video indexing task for each upload and monitors the task status until completion or failure:

1 for video_file in video_files:
2     video_filename = os.path.basename(video_file)
3     if video_filename in existing_filenames:
4         print(
5             f"File '{video_filename}' is already uploaded. Skipping to the next video."
6         )
7         continue
8 
9     try:
10         print(f"Uploading {video_filename}")
11         # Assuming task creation and upload logic remains the same
12         task = client.task.create(index_id=INDEX_ID, file=video_file, language="en")
13         print(f"Task id={task.id}")
14 
15         def on_task_update(task: Task):
16             print(f"  Status={task.status}")
17 
18         task.wait_for_done(callback=on_task_update)
19 
20         if task.status != "ready":
21             raise RuntimeError(f"Indexing failed with status {task.status}")
22 
23         print(
24             f"Uploaded {video_filename}. The unique identifier of your video is {task.video_id}."
25         )
26 
27     except RuntimeError as e:
28         print(f"Error uploading {video_filename}: {e}")
29         # Continue with the next video

The search_twelve_labs function invokes the POST method of the /search endpoint with specific parameters and a query provided by the user:

1 def search_twelve_labs(query):
2     url = "https://api.twelvelabs.io/v1.2/search"
3     payload = {
4         "search_options": ["visual", "conversation"],
5         "adjust_confidence_level": 0.5,
6         "group_by": "clip",
7         "threshold": "low",
8         "sort_option": "score",
9         "operator": "or",
10         "conversation_option": "semantic",
11         "page_limit": 50,
12         "query": query,  # Use the input query here
13         "index_id": os.getenv("TL_INDEX")  # Replace with your actual index ID
14     }
15     headers = {
16         "accept": "application/json",
17         "x-api-key": os.getenv("TL_API_KEY"),  # Replace with your actual API key
18         "Content-Type": "application/json"
19     }
20     
21     response = requests.post(url, json=payload, headers=headers)
22     
23     if response.status_code == 200:
24         return response.json()
25     else:
26         return {"error": "Request failed", "status_code": response.status_code, "message": response.text}

Cricket Video Analyzer

Summary: “Cricket Video Analyzer” combines the capabilities of the TwelveLabs Video Understanding Platform with language models to offer comprehensive insights into a batter’s performance. It focuses on analyzing Glenn Maxwell of Australia as a case study.
Description: The application has been developed to meet the growing demand for sophisticated video analysis in cricket. Key features include:

Video understanding: Uses the TwelveLabs API to analyze cricket matches, extracting detailed information about the batter’s playing style and techniques.
Data augmentation: Integrates ball-by-ball commentary data from ESPN cricinfo APIs to enhance the video analysis with textual descriptions of the gameplay.
Vector store creation: Builds a vector store from the combined dataset of video analysis and ball-by-ball commentary, enabling efficient information retrieval.
AI-powered query answering: Utilizes GPT-4 in a Retrieval-Augmented Generation (RAG) setup to answer complex questions about a batter’s performance and style based on the analyzed data.

GitHub: Cricket video analyzer
Demo: Cricket Batsman Analyzer

The application was developed by Raghavan Muthuregunathan

Integration with TwelveLabs

The code below automates the process of extracting detailed batting information from multiple videos, creating a dataset for further analysis. It invokes the generate.text method of the TwelveLabs Python SDK to generate a ball-by-ball commentary for Maxwell’s cricket innings. It compiles all the generated commentaries into a single text file named “maxwell_innings_video_analysis_concat.”

1 video_ids = ["66370d91d1cd5a287c957d14", "66370d91d1cd5a287c957d15", "66370d95d1cd5a287c957d16","663715d0d1cd5a287c957d17"]
2 
3 summary = ""
4 
5 for id in video_ids:
6     print(id)
7     res = client.generate.text(video_id= id,prompt=" Listen to the commentary. Generate a ball by ball commentary of how Maxwell played. For  every ball, generate a description of the shot that Maxwell played. if Maxwell got out, explain how he got out?")
8     summary += res.data
9     print(len(summary))
10 
11 with open("maxwell_innings_video_analysis_concat.txt", "w") as w:
12     w.write(summary)

Ready Player N

Summary: Ready Player N is a Streamlit-based application that helps you discover unique features in hiking trails by analyzing hiking videos using the Marengo video understanding engine.
Description: Ready Player N streamlines the discovery of special attractions along hiking trails, such as waterfalls. The application indexes a set of hiking videos, allowing you to search for relevant information. In order to improve search accuracy, the queries are refined using Groq. In response, the application provides the following:

A thumbnail of the matching video segment.
A description of the featured attraction.
A link to a hiking website with additional details.
The ability to watch the matching video segment directly within the integrated player.

The application was developed by Bryce Hirschfeld, James Beasley, Matt Florence, Drakosfire Meigs, and Nooshin Hashemi.

SportsSnap

Summary: SportsSnap is a browser extension that enhances the sports viewing experience by providing real-time player, team, and game information overlays. It uses the TwelveLabs Video Understanding Platform for multimodal search.
Description: The application bridges the gap between video content and real-time sports data, offering instant access to detailed information while watching game highlights or replays.

The application was developed by Axel VandenHeuvel and Navan Chauhan.

Key features include:

Real-time information overlay on sports videos
Player and team statistics integration
Game situation analysis
Multimodal search capabilities

Integration with TwelveLabs

The integration with the TwelveLabs Video Understanding Platform provides the following functionalities:

Video understanding: Extract and analyze visual and audio information from sports content, improving the accuracy of video identification and content analysis.
Advanced search capabilities: Perform complex queries across various data types (visual, audio, and textual) to enhance the app’s ability to provide relevant information to users.
Improved video-to-game matching: Improve the accuracy of matching videos to specific games in ESPN’s database, addressing one of the main challenges faced during development.
Enhanced user experience: Provide users with more accurate and contextually relevant information, creating a more seamless and informative viewing experience.

SnowRivals

Summary: SnowRivals is an AI-powered application for skiers and snowboarders that analyzes trick videos, provides improvement feedback, and aligns with competition judging criteria. The application offers detailed insights and suggestions for enhancing trick execution and scoring.

Description: SnowRivals uses TwelveLabs’ video summarization and embedding technology to analyze snowboarding and skiing trick videos by extracting the following structured data:

Trick identification
Athlete performance
Success and failure detection
Timing information

The application was developed by Jeremy London, Francesco Vassalli, Neil Fonseca, and Reid Rumack.

SnowRivals was one of the winners at the AI Tinkerers Multimodal Hackathon .

GitHub: multimodal-ui and MultiModal.

Integration with TwelveLabs

The generate function uses the Generate API suite to analyze a snowboarding competition video. It sends a POST request to the /summarize endpoint with the following parameters:

video_id: Identifier of the video to analyze.
type: Set to “summary”.
prompt: Detailed instructions for generating a table of snowboarding tricks. It includes a predefined list of trick descriptions to aid in trick identification.
temperature: Controls randomness in the output (set to 0.5).

If successful, the function returns a summary table containing the following:

Athlete: Identifier for the person performing each trick
Trick: Name of the trick performed
Result: Success or failure of the trick
Half: Presence of a half pipe (“Yes”/“No”)
Rail: Presence of a rail (“Yes”/“No”)

If the summary generation fails, the function prints an error message with the video_id and the API response.

1 def generate(video_id:str)->str:
2     BASE_URL = "https://api.twelvelabs.io/v1.2"
3     trick_descriptions = '''Indy: Grabbing the toe edge of the board between the bindings with the back hand.
4 Mute: Grabbing the toe edge of the board between the bindings with the front hand.
5 Stalefish: Grabbing the heel edge of the board between the bindings with the back hand.
6 Melon: Grabbing the heel edge of the board between the bindings with the front hand.
7 Tail Grab: Grabbing the tail of the board.
8 Nose Grab: Grabbing the nose of the board.
9 Method: Grabbing the heel edge of the board with the front hand while arching the back and extending the legs.
10 Japan: Grabbing the toe edge of the board with the front hand while tucking the knees and rotating the board.
11 Seatbelt: Grabbing the nose of the board with the back hand.
12 Truck Driver: Grabbing the nose of the board with the front hand and the tail with the back hand simultaneously.
13 Rodeo: A backward flip with a 180 or 540-degree spin.
14 Misty: A forward flip with a 180 or 540-degree spin.
15 Cork: An off-axis spin, where the boarder is tilted.
16 Double Cork: A double off-axis spin.
17 Wildcat: A backflip with a rotation around the snowboarder’s side.
18 Jumps
19 Ollie: Lifting the front foot, followed by the back foot, to jump.
20 Nollie: Lifting the back foot, followed by the front foot, to jump.
21 Slides and Grinds
22 Boardslide: Sliding with the board perpendicular to the rail.
23 Lipslide: Approaching from the opposite side and sliding with the board perpendicular to the rail.
24 50-50: Sliding straight along a rail with the board parallel.
25 Nose Slide: Sliding on the nose of the board.
26 Tail Slide: Sliding on the tail of the board.
27 Blunt Slide: Sliding on the tail of the board with the nose raised.
28 Buttering
29 Nose Butter: Pressing down on the nose of the board while rotating the tail.
30 Tail Butter: Pressing down on the tail of the board while rotating the nose.
31 Nose Roll: A 180-degree rotation while buttering on the nose.
32 Tail Roll: A 180-degree rotation while buttering on the tail.
33 '''
34 
35 
36     data = {
37         "video_id": video_id,
38         "type": "summary",
39         "prompt": f"This video has a snowboarding competition. Use expert Generate a table that records one row per trick performed. Add one column called Athlete to label which person performed each trick. If you do not know their names just refer to them as person-1 or person-2 as they appear sequentially in the video. In a second column called Trick, name the trick. In the third column called Result note if the trick was a Success or a Failure. In a fourth column called Half say Yes if the video has a half pipe and No if it does not. In a fifth column called Rail say Yes if the video has a rail and no if it does not. Keep your response brief and do not include anything aside from the table. If you are unsure of what to call a track you may reference this vocabulary list {trick_descriptions}",
40         "temperature": 0.5
41     }
42 
43     response = requests.post(f"{BASE_URL}/summarize", json=data, headers={"x-api-key": key})
44     response_dict = json.loads(response.text)
45     if 'summary' in response_dict:
46         return response_dict['summary']
47     else:
48         print(f"Summary for {video_id} unable to be generated. Got response {response}")

Smart Sweat AI

Summary: Smart Sweat AI is a mobile application that uses artificial intelligence to provide real-time feedback on exercise form during workouts. It uses the device’s camera to analyze your movements and offer personalized corrections and improvements.

Description: The application serves as a digital personal trainer, utilizing AI technology to:

Analyze exercise form in real-time through your phone camera
Provide instant feedback and corrections on posture and movement
Offer a comprehensive library of exercises
Track your progress and personalize workout experiences

Smart Sweat AI uses the TwelveLabs Generate API suite to provide advanced video analysis and personalized fitness instruction.

The application was developed by Jesse Neumann, Hannah Neumann, and Roblynn Neumann.

Smart Sweat AI was one of the winners at the AI Tinkerers Multimodal Hackathon .

Website: Smart Sweat AI

National Park Advisor

Summary: National Park Advisor is an application designed to assist you in researching and planning trips to National Parks.

Description: The application utilizes two distinct RAG pipelines:

Park information pipeline: Retrieves and presents detailed information about specific National Parks. It uses a data store to search for park information efficiently. When the data store does not have sufficient data for a particular park, the application supplements it by searching the internet, ensuring comprehensive coverage.
Trip planning pipeline: Assists in planning visits to National Parks.

Key features include:

National Park research assistance
Trip planning for National Parks
Video content search and classification
Transcript extraction from videos
Internet search capability for supplemental park information

The application utilizes:

TwelveLabs Python SDK: Video search, transcript extraction, and content classification.
Pinecone: Stores data.
Tavily: Internet search for additional content.
Langgraph: Workflow management.
Groq: Language model (LLM) for fast inference.

The application was developed by Sarah Kainec and Lexie Marinelli.

GitHub: ntl-park-advisor
Demo: National Park Advisor

Integration with TwelveLabs

The get_videos_from_twelve function performs search requests and retrieves transcriptions for up to four videos that match the provided query. It returns a dictionary that includes all key-value pairs from the state parameter, to which it adds the video URLs and transcription data:

1 def get_videos_from_twelve(state: dict) -> dict:
2    print("getting videos")
3    load_dotenv()
4    client = TwelveLabs(api_key=os.getenv("TL_API_KEY"))
5 
6    search_results = client.search.query(
7        index_id=TL_INDEX_ID,
8        query_text=state["query"],
9        options=["visual"]
10    )
11 
12    filtered_search = []
13    unique_vids_id = []
14    for clips in search_results.data.root:
15        if clips.video_id not in unique_vids_id:
16            unique_vids_id.append(clips.video_id)
17            filtered_search.append(clips)
18 
19    transcript_data = []
20    urls = []
21    for id in unique_vids_id[0:4]:
22        script = client.index.video.transcription(
23            index_id=TL_INDEX_ID,
24            id=id
25        )
26        url = client.index.video.retrieve(index_id=TL_INDEX_ID, id=id)
27        urls.append(url)
28        whole_script = ""
29        for vid_value in script.root:
30            whole_script = whole_script + " " + vid_value.value
31 
32        transcript_data.append({"video_id": id, "video_url": url.hls.video_url, "transcript": whole_script})
33 
34    return {
35        **state,
36        "video_urls": [url.hls.video_url for url in urls],
37        "transcript_data": transcript_data
38    }

The get_classified_videos_from_twelve function classifies videos based on predefined, filters for the specified destination, and returns an updated state dictionary with URLs for up to four matching videos.

1 classification = [
2    {
3        "name": "Rocky Mountain National Park",
4        "prompts": [
5            "Things to do in Rocky Mountain National Park",
6            "Learn about Rocky Mountain National Park",
7            "Prepare for a trip to Rocky Mountain National Park",
8            "RMNP",
9            "Colorado"
10        ]
11    },
12    {
13        "name": "Glacier National Park",
14        "prompts": [
15            "Things to do in Glacier National Park",
16            "Wildlife at Glacier National Park",
17            "Ecology of Glacier National Park",
18            "Glacier Science",
19            "Montana"
20        ]
21    }
22 ]
23 
24 
25 def get_classified_videos_from_twelve(state: dict) -> dict:
26    load_dotenv()
27    client = TwelveLabs(api_key=os.getenv("TL_API_KEY"))
28 
29    classified_result = client.classify.index(
30        index_id=TL_INDEX_ID,
31        options=["visual"],
32        classes=classification,
33    )
34 
35    filtered_search = []
36    unique_vids_id = []
37    for clips in classified_result.data.root:
38        if clips.video_id not in unique_vids_id and clips.classes.root[0].name == state["destination"]:
39            unique_vids_id.append(clips.video_id)
40            filtered_search.append(clips)
41    urls = []
42    for id in unique_vids_id[0:4]:
43        url = client.index.video.retrieve(index_id=TL_INDEX_ID, id=id)
44        urls.append(url)
45 
46    return {
47        **state,
48        "video_urls": [url.hls.video_url for url in urls],
49    }

1	for video_file in video_files:
2	video_filename = os.path.basename(video_file)
3	if video_filename in existing_filenames:
4	print(
5	f"File '{video_filename}' is already uploaded. Skipping to the next video."
6	)
7	continue
8
9	try:
10	print(f"Uploading {video_filename}")
11	# Assuming task creation and upload logic remains the same
12	task = client.task.create(index_id=INDEX_ID, file=video_file, language="en")
13	print(f"Task id={task.id}")
14
15	def on_task_update(task: Task):
16	print(f" Status={task.status}")
17
18	task.wait_for_done(callback=on_task_update)
19
20	if task.status != "ready":
21	raise RuntimeError(f"Indexing failed with status {task.status}")
22
23	print(
24	f"Uploaded {video_filename}. The unique identifier of your video is {task.video_id}."
25	)
26
27	except RuntimeError as e:
28	print(f"Error uploading {video_filename}: {e}")
29	# Continue with the next video

1	def search_twelve_labs(query):
2	url = "https://api.twelvelabs.io/v1.2/search"
3	payload = {
4	"search_options": ["visual", "conversation"],
5	"adjust_confidence_level": 0.5,
6	"group_by": "clip",
7	"threshold": "low",
8	"sort_option": "score",
9	"operator": "or",
10	"conversation_option": "semantic",
11	"page_limit": 50,
12	"query": query, # Use the input query here
13	"index_id": os.getenv("TL_INDEX") # Replace with your actual index ID
14	}
15	headers = {
16	"accept": "application/json",
17	"x-api-key": os.getenv("TL_API_KEY"), # Replace with your actual API key
18	"Content-Type": "application/json"
19	}
20
21	response = requests.post(url, json=payload, headers=headers)
22
23	if response.status_code == 200:
24	return response.json()
25	else:
26	return {"error": "Request failed", "status_code": response.status_code, "message": response.text}

1	video_ids = ["66370d91d1cd5a287c957d14", "66370d91d1cd5a287c957d15", "66370d95d1cd5a287c957d16","663715d0d1cd5a287c957d17"]
2
3	summary = ""
4
5	for id in video_ids:
6	print(id)
7	res = client.generate.text(video_id= id,prompt=" Listen to the commentary. Generate a ball by ball commentary of how Maxwell played. For every ball, generate a description of the shot that Maxwell played. if Maxwell got out, explain how he got out?")
8	summary += res.data
9	print(len(summary))
10
11	with open("maxwell_innings_video_analysis_concat.txt", "w") as w:
12	w.write(summary)

1	def generate(video_id:str)->str:
2	BASE_URL = "https://api.twelvelabs.io/v1.2"
3	trick_descriptions = '''Indy: Grabbing the toe edge of the board between the bindings with the back hand.
4	Mute: Grabbing the toe edge of the board between the bindings with the front hand.
5	Stalefish: Grabbing the heel edge of the board between the bindings with the back hand.
6	Melon: Grabbing the heel edge of the board between the bindings with the front hand.
7	Tail Grab: Grabbing the tail of the board.
8	Nose Grab: Grabbing the nose of the board.
9	Method: Grabbing the heel edge of the board with the front hand while arching the back and extending the legs.
10	Japan: Grabbing the toe edge of the board with the front hand while tucking the knees and rotating the board.
11	Seatbelt: Grabbing the nose of the board with the back hand.
12	Truck Driver: Grabbing the nose of the board with the front hand and the tail with the back hand simultaneously.
13	Rodeo: A backward flip with a 180 or 540-degree spin.
14	Misty: A forward flip with a 180 or 540-degree spin.
15	Cork: An off-axis spin, where the boarder is tilted.
16	Double Cork: A double off-axis spin.
17	Wildcat: A backflip with a rotation around the snowboarder’s side.
18	Jumps
19	Ollie: Lifting the front foot, followed by the back foot, to jump.
20	Nollie: Lifting the back foot, followed by the front foot, to jump.
21	Slides and Grinds
22	Boardslide: Sliding with the board perpendicular to the rail.
23	Lipslide: Approaching from the opposite side and sliding with the board perpendicular to the rail.
24	50-50: Sliding straight along a rail with the board parallel.
25	Nose Slide: Sliding on the nose of the board.
26	Tail Slide: Sliding on the tail of the board.
27	Blunt Slide: Sliding on the tail of the board with the nose raised.
28	Buttering
29	Nose Butter: Pressing down on the nose of the board while rotating the tail.
30	Tail Butter: Pressing down on the tail of the board while rotating the nose.
31	Nose Roll: A 180-degree rotation while buttering on the nose.
32	Tail Roll: A 180-degree rotation while buttering on the tail.
33	'''
34
35
36	data = {
37	"video_id": video_id,
38	"type": "summary",
39	"prompt": f"This video has a snowboarding competition. Use expert Generate a table that records one row per trick performed. Add one column called Athlete to label which person performed each trick. If you do not know their names just refer to them as person-1 or person-2 as they appear sequentially in the video. In a second column called Trick, name the trick. In the third column called Result note if the trick was a Success or a Failure. In a fourth column called Half say Yes if the video has a half pipe and No if it does not. In a fifth column called Rail say Yes if the video has a rail and no if it does not. Keep your response brief and do not include anything aside from the table. If you are unsure of what to call a track you may reference this vocabulary list {trick_descriptions}",
40	"temperature": 0.5
41	}
42
43	response = requests.post(f"{BASE_URL}/summarize", json=data, headers={"x-api-key": key})
44	response_dict = json.loads(response.text)
45	if 'summary' in response_dict:
46	return response_dict['summary']
47	else:
48	print(f"Summary for {video_id} unable to be generated. Got response {response}")

1	def get_videos_from_twelve(state: dict) -> dict:
2	print("getting videos")
3	load_dotenv()
4	client = TwelveLabs(api_key=os.getenv("TL_API_KEY"))
5
6	search_results = client.search.query(
7	index_id=TL_INDEX_ID,
8	query_text=state["query"],
9	options=["visual"]
10	)
11
12	filtered_search = []
13	unique_vids_id = []
14	for clips in search_results.data.root:
15	if clips.video_id not in unique_vids_id:
16	unique_vids_id.append(clips.video_id)
17	filtered_search.append(clips)
18
19	transcript_data = []
20	urls = []
21	for id in unique_vids_id[0:4]:
22	script = client.index.video.transcription(
23	index_id=TL_INDEX_ID,
24	id=id
25	)
26	url = client.index.video.retrieve(index_id=TL_INDEX_ID, id=id)
27	urls.append(url)
28	whole_script = ""
29	for vid_value in script.root:
30	whole_script = whole_script + " " + vid_value.value
31
32	transcript_data.append({"video_id": id, "video_url": url.hls.video_url, "transcript": whole_script})
33
34	return {
35	**state,
36	"video_urls": [url.hls.video_url for url in urls],
37	"transcript_data": transcript_data
38	}

1	classification = [
2	{
3	"name": "Rocky Mountain National Park",
4	"prompts": [
5	"Things to do in Rocky Mountain National Park",
6	"Learn about Rocky Mountain National Park",
7	"Prepare for a trip to Rocky Mountain National Park",
8	"RMNP",
9	"Colorado"
10	]
11	},
12	{
13	"name": "Glacier National Park",
14	"prompts": [
15	"Things to do in Glacier National Park",
16	"Wildlife at Glacier National Park",
17	"Ecology of Glacier National Park",
18	"Glacier Science",
19	"Montana"
20	]
21	}
22	]
23
24
25	def get_classified_videos_from_twelve(state: dict) -> dict:
26	load_dotenv()
27	client = TwelveLabs(api_key=os.getenv("TL_API_KEY"))
28
29	classified_result = client.classify.index(
30	index_id=TL_INDEX_ID,
31	options=["visual"],
32	classes=classification,
33	)
34
35	filtered_search = []
36	unique_vids_id = []
37	for clips in classified_result.data.root:
38	if clips.video_id not in unique_vids_id and clips.classes.root[0].name == state["destination"]:
39	unique_vids_id.append(clips.video_id)
40	filtered_search.append(clips)
41	urls = []
42	for id in unique_vids_id[0:4]:
43	url = client.index.video.retrieve(index_id=TL_INDEX_ID, id=id)
44	urls.append(url)
45
46	return {
47	**state,
48	"video_urls": [url.hls.video_url for url in urls],
49	}