The example projects on this page demonstrate using the Twelve Labs Video Understanding Platform to enhance educational experiences. These projects address complex topics in education and pave the way for the future of AI-powered e-learning solutions.

AI'm Right

Summary: AI'm Right is an educational application developed by Muhammad Adil Fayya during SBHacks’25 at the University of California, Santa Barbara. It leverages artificial intelligence to streamline studying through quiz generation, lecture summarization, and video content search capabilities.

Description: The application integrates multiple AI services through a Flask backend that manages the processing operations and a Streamlit frontend that provides the user interface.:

Twelve Labs for video understanding
Claude for content processing,
Aryn for additional functionality

These services are integrated through a Flask backend that manages the processing operations and a Streamlit frontend that provides the user interface.

GitHub repo: darshanrao/RAG-Enhanced-Learning-Assistant

Integration with TwelveLabs

AI'm Right integrates with the Labs Video Understanding Platform for searching and retrieving video content, enabling you to locate specific topics or concepts within your recorded lectures without manually scanning through entire videos.

The query function uses the Twelve Labs Python SDK to perform one or more text queries using both the visual and audio search options. For each query, it retrieves paginated search results that include video IDs, confidence scores, and timestamp information. The function can optionally save these results to a JSON file and returns them as a nested list of dictionaries.

def query(self, query_text_list, file_name=None):
   all_results = []  # List of lists of dictionaries


   for query_text in query_text_list:
       search_results = self.client.search.query(
           index_id=os.getenv("INDEX_ID"),
           query_text=query_text,
           # query={"text": query_text},
          
           # query = {
           #     "$not": {
           #         "origin": {
           #             "text": query_text
           #         },
           #         "sub": {
           #             "$or": [
           #                 {
           #                     "text": "News Anchor"
           #                 },
           #                 {
           #                     "text": "Podcast"
           #                 }
           #             ]
           #         }
           #     }
           # },


           options=["visual","audio"]
       )
      
       query_result = []  # List of dictionaries for this query
       while True:
          
           try:
               # Retrieve each page of search results
               page_data = next(search_results)
               for clip in page_data:  # Limiting to 3 results per page
                   query_result.append({
                       "video_id": clip.video_id,
                       "score": clip.score,
                       "start": clip.start,
                       "end": clip.end,
                       "confidence": clip.confidence,
                       # "metadata": clip.metadata,
                   })
           except StopIteration:
               break
       all_results.append(query_result)  # Append this query's results as a list


   if file_name:
       self.save_json(all_results, file_name)


   return all_results  # List of lists of dictionaries

The get_video_info function retrieves detailed information about a specific video using its ID. It returns the complete video metadata, including the HLS streaming URL.

def get_video_info(video_id):
   api_key= os.getenv("TWELVE_LABS_KEY")


   index_id=os.getenv("INDEX_ID")
   url = f"https://api.twelvelabs.io/v1.3/indexes/{index_id}/videos/{video_id}"
   headers = {
       "x-api-key": api_key,
       "Content-Type": "application/json",
   }
  
   response = requests.get(url, headers=headers)


   if response.status_code == 200:
       video_info = response.json()  # Parse response as JSON
       return video_info
   elif response.status_code == 400:
       print("The request has failed. Check your parameters.")
       return None
   else:
       print(f"Error: Status code {response.status_code}")
       return None

NeuroLearn

Summary: NeuroLearn is an AI-based learning platform that integrates neurofeedback to personalize education by creating.

Description: The application analyzes brainwave data to select and emphasize parts of lectures that match the student's educational objectives. This approach keeps students engaged and improves their learning results. Developed by Akhil Dhavala, Ayush Khandelwal, Jackson Mowatt Gok, and Jacky Wong, NeuroLearn won 1st Place at the TEDAI Multimodal Hackathon (23 Labs) in San Francisco.

GitHub repo: NeuroLearn

Integration with Twelve Labs

NeuroLearn uses the Twelve Labs Video Understanding Platform to create custom snippets from lectures tailored to each student's knowledge, background, and interests.

The code below indexes videos from YouTube and for each video generates a summary and highlights:

    def __init__(self):
        self.api_url = os.getenv("TWELVE_LABS_BASE_URL")
        self.headers = {"x-api-key": os.getenv("TWELVE_LABS_API_KEY")}

    def index_youtube_video(self, index_id: str, youtube_url: str):
        task_url = f"{self.api_url}/tasks/external-provider"
        data = {
            "index_id": index_id,
            "url": youtube_url,
        }
        response = requests.post(task_url, headers=self.headers, json=data)
        # you can get the video ID from video_id
        return response.json()

    def highlight_video(self, body: HighlightVideoBody) -> HighlightVideoResponse:
        url = f"{self.api_url}/summarize"
        data = body.dict()
        response = requests.post(url, headers=self.headers, json=data)
        return HighlightVideoResponse(**response.json())

    def summarize(self, body: HighlightVideoBody) -> HighlightVideoResponse:
        url = f"{self.api_url}/summarize"
        data = body.dict()
        response = requests.post(url, headers=self.headers, json=data)
        return response.json()

42Labs - Personalized Podcast Builder

Summary: The 42 Labs Personalized Podcast Builder transforms how you learn on the go by synthesizing high-quality content from diverse sources like TED talks, podcasts, and articles into customized podcasts.

Description: The application analyzes your preferences, including topics of interest and proficiency levels, to curate content in various languages. You can interactively refine your learning experience by selecting subtopics and providing feedback, enhancing the application's ability to offer relevant material. The application not only overcomes language and learning barriers but also holds the potential to evolve into a tool for creating personalized educational videos. The application was developed by Shivani Poddar, David Salib, Varun Theja, and Everett Knag.

GitHub:

Frontend: TEDAI_2023.
Backend: 42labs-answer-to-everything.

Integration with Twelve Labs

The application uses the /classify/bulk endpoint to identify videos relevant to a specified topic:

def classify_videos(index_id, sub_topic, api_key):
    CLASSIFY_BULK_URL = f"{url}/classify/bulk"

    data =  {
    "options": ["conversation", "text_in_video"],
    "index_id": index_id,
    "classes": [{"name": sub_topic,
        "prompts": [
            sub_topic,
        ]}]
    }
    headers = {
        "accept": "application/json",
        "Content-Type": "application/json",
        "x-api-key": api_key
    }

    response = requests.post(CLASSIFY_BULK_URL, headers=headers, json=data)
    print (f'Status code: {response.status_code}')
    print(response.json())
    return response.json()["data"]

For each relevant video, the application invokes the /summarize endpoint to summarize videos as lists of chapters:

def summarize_video(video_id, api_key):
    payload = {
        "type": "chapter",
        "video_id": video_id,
    }
    headers = {
        "accept": "application/json",
        "x-api-key": api_key,
        "Content-Type": "application/json"
    }

    url = "https://api.twelvelabs.io/v1.2/summarize"
    response = requests.post(url, json=payload, headers=headers)

    print(response.text)
    chapter_summaries = [x["chapter_summary"] for x in response.json()["chapters"]]
    return chapter_summaries