Combined queries allow you to search for different elements spanning across multiple sources of information. A combined query can include any number of subqueries linked with any number of operators. Combined queries are executed in one API request and offer the following main benefits:

Enhanced flexibility: By specifying multiple conditions, you can build combined queries that, depending on your needs, are either very specific or broad.
Improved precision: Combined queries allow you to be specific about what you want to find in your videos, which can help you retrieve only the most relevant results.

Prerequisites

The examples in the following sections assume the following:

You’re familiar with the concepts that are described on the Platform overview page.
You've already created an index, and the Marengo video understanding engine is enabled for this index.
You've already uploaded a video, and the platform has finished indexing it.

The basics of combined queries

The most important parameters that you specify when performing a search request using combined queries are described below:

index_id: The unique identifier of the index to search.
search_options: An array of strings specifying the source of information the platform uses when performing a search. The following values are supported: visual, conversation, text_in_video, and logo. For details, see the Search options page.
(Optional) conversation_option: A string that specifies the type of search the platform will perform. The following values are supported: semantic, exact_match. For details, see the Conversation option page.
query: An object that defines a combined query. In its simplest form, the query object is composed of the following key-value pairs:
- text: A string representing your search terms.
- (Optional)search_options: If set, overrides the search options for this query.
- (Optional) conversation_option: If set, overrides the conversation option for this query.

The response is a SearchResult object, which contains the following fields:

pool: A SearchPool object that contains the total number of videos within the index, the total duration of the videos, and the unique identifier of the index that you've searched.
data: A list of SearchData objects, each of which contains the following fields:
- score: A quantitative value determined by the AI engine representing the level of confidence that the results match your search terms.
- start: The start time of the matching video clip, expressed in seconds.
- end: The end time of the matching video clip, expressed in seconds.
- video_id: The unique identifier of the video that matched your search terms.
- confidence: A qualitative indicator based on the value of the score field. This field can take one of the following values:
  - high
  - medium
  - low
  - extremely_low
- thumbnail_url: If thumbnail generation has been enabled for this index, the platform returns a string representing the URL of the thumbnail of the segment. Note that the URL expires in one hour.
page_info: A SearchPageInfo object that provides information about pagination. The platform returns the results one page at a time, with a default limit of 10 results per page. For details about retrieving the subsequent pages of results and configuring the pagination behavior, see the Pagination > Search results page.

For a description of each field in the request and response, see the API Reference > Make a combined search request page.

The following example searches for car accidents based on visual cues:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>",
  query={
    "text": "<YOUR_QUERY>"
  },
  options=["visual"],
)
# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>'
  query: {
    text: '<YOUR_QUERY>',
  },
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
    );
  });
}

The following output was truncated for brevity:

video_id=639963a1ce36463e0199c8c7 score=81.79 start=117 end=125 confidence=medium
video_id=639963a1ce36463e0199c8c7 score=75.61 start=420 end=430 confidence=medium
video_id=63a09373ce36463e0199c8de score=72.9 start=15 end=18 confidence=low

Use operators

The platform allows you to build combined queries using the following operators:

and: A dictionary where the key is the $and string, and the value is an array of objects. Each object is a subquery. The platform returns the video clips for which all the specified queries match.
or: A dictionary where the key is the $or string, and the value is an array of objects. Each object is a subquery. The platform returns the video clips for which at least one of the queries matches.
not: A dictionary where the key is the $not string and the value is a dictionary composed of two queries named origin and sub. The platform returns the video clips that match the origin query but do not match the sub query. Note that the origin and sub queries can include any number of subqueries.
then: A dictionary where the key is the $then string, and the value is an array of objects. Each object is a subquery. The platform will return only the results for which the order of the matching video clips is the same as the order of your queries.

The following is an example of a combined query that finds the moments in your videos where a blue or a red car appears:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>",
  query={
    "$or": [
      {
        "text": "red car",
      },
      {
        "text": "blue car",
      }
    ]
  },
  options=["visual"],
)

# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

  import { TwelveLabs, SearchData } from 'twelvelabs-js';

  const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});  
  
  let searchResults = await client.search.query({
    indexId: '<YOUR_INDEX_ID>',
    query: {
      $or: [
        {
          text: 'red car',
        },
        {
          text: 'blue car',
        },
      ],
    },
    options: ['visual'],
  });
  printPage(searchResults.data);
  while (true) {
    const page = await searchResults.next();
    if (page === null) break;
    else printPage(page);
  }
  // Utility function to print a specific page
  function printPage(searchData) {
    (searchData as SearchData[]).forEach((clip) => {
      console.log(
        `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
      );
    });

The following output was truncated for brevity:

 video_id=65d60bcf48db9fa780cb415e score=64.6 start=191.0 end=192.0 confidence=low
 video_id=65d6131c48db9fa780cb415f score=64.6 start=191.0 end=192.0 confidence=low
 video_id=65d60bcf48db9fa780cb415e score=64.6 start=368.0 end=369.0 confidence=low
 video_id=65d6131c48db9fa780cb415f score=64.6 start=368.0 end=369.0 confidence=low

The following is an example of a combined query that finds the moments in your video where:

Someone is cooking
Italian food is mentioned in the conversation
Neither the word spaghetti is displayed on the screen nor lasagna is mentioned in the conversation.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query={
        "$not": {
            "origin": {
                "$and": [
                    {
                        "text": "Someone is cooking",
                    },
                    {
                        "text": "Italian food",
                        "search_options": ["conversation"]
                    }
                ]
            },
            "sub": {
                "$or": [
                    {
                        "text": "spaghetti",
                        "search_options": ["text_in_video"]
                    },
                    {
                        "text": "lasagna",
                        "search_options": ["conversation"]
                    }
                ]
            }
        }
    },
    options=["visual"],
)

# Utility function to print a specific page


def print_page(page):
    for clip in page:
        print(
            f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
        )


print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: {
    $not: {
      origin: {
        $and: [
          {
            text: 'Someone is cooking',
          },
          {
            text: 'Italian food',
            search_options: ['conversation'],
          },
        ],
      },
      sub: {
        $or: [
          {
            text: 'spaghetti',
            search_options: ['text_in_video'],
          },
          {
            text: 'lasagna',
            search_options: ['conversation'],
          },
        ],
      },
    },
  },
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
    );
  });
}

The following output was truncated for brevity:

video_id=65d6131c48db9fa780cb415f score=92.28 start=25.0 end=36.0 confidence=high
video_id=65d5fbad48db9fa780cb415c score=84.39 start=408.0 end=438.0 confidence=high
video_id=65d5fbad48db9fa780cb415c score=83.85 start=22.0 end=47.0 confidence=high

In the example code above, note that the value of the search_options parameter is set to ["visual"] for the entire query, and it is overridden on a per-subquery basis.

Advanced techniques

This section illustrates some advanced techniques you can use to extend the functionality of combined queries.

Time-proximity search

When building combined queries, you can use the proximity parameter to extend the lower and upper boundaries of each subquery. This parameter is expressed in seconds.

Note that the proximity parameter can only be used when your combined query is composed of two or more subqueries.

The following example code uses the and operator and the proximity parameter to find all car accidents that happened within 30 seconds before someone wins a race:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query={
        "$and": [
          {
              "text": "winning the race"
          },
            {
              "text": "car accident"
          }
        ],
        "proximity": 30
    },
    options=["visual"],
)

# Utility function to print a specific page

def print_page(page):
    for clip in page:
        print(
            f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
        )


print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: {
    $and: [
      {
        text: 'winning the race',
      },
      {
        text: 'car accident',
      },
    ],
    proximity: 30,
  },
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
    );
  });
}

The following output was truncated for brevity:

video_id=65d5fbad48db9fa780cb417a score=83.7 start=283.0 end=325.0 confidence=high
video_id=65d5fbad48db9fa780cb417a score=83.69 start=342.0 end=377.0 confidence=high
video_id=65d60bcf48db9fa780cb417a score=83.62 start=323.0 end=334.0 confidence=high

Specify the order of the matching video clips

The example code below uses the then operator to find the moments in your videos where the following occur in the specified order:

A player makes a long pass.
A player dribbles past a defender.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query={
        "$then": [
            {
                "text": "Player is making a long pass"
            },
            {
                "text": "Player is dribbling past a defender"
            }
        ]
    },
    options=["visual"],
)

# Utility function to print a specific page
def print_page(page):
    for clip in page:
        print(
            f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
        )


print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: {
    $then: [
      {
        text: 'Player is making a long pass',
      },
      {
        text: 'Player is dribbling past a defender',
      },
    ],
  },
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
    );
  });
}

The following example output was truncated for brevity:

video_id=65d60bcf48db9fa780cb412a score=83.22 start=419.0 end=444.0 confidence=high
video_id=65d5fbad48db9fa780cb412a score=82.75 start=47.0 end=75.0 confidence=medium
video_id=65d60bcf48db9fa780cb412a score=78.21 start=294.0 end=304.0 confidence=medium

The example code below combines the then and or operators to find all the moments in your videos where the following occur in the specified order:

A player makes a long pass or the words "amazing pass" are mentioned in the conversation
A player dribbles past a defender.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query={
        "$then": [
            {
                "$or": [
                    {
                        "text": "Player makes a long pass"
                    },
                    {
                        "text": "Amazing pass",
                        "search_options": ["conversation"]
                    }
                ]
            },
            {
                "text": "Player dribbles past a defender"
            }
        ]
    },
    options=["visual"],
)

# Utility function to print a specific page
def print_page(page):
    for clip in page:
        print(
            f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
        )


print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: {
    $then: [
      {
        $or: [
          {
            text: 'Player makes a long pass',
          },
          {
            text: 'Amazing pass',
            search_options: ['conversation'],
          },
        ],
      },
      {
        text: 'Player dribbles past a defender',
      },
    ],
  },
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
    );
  });
}

The following example output was truncated for brevity:

video_id=65d60bcf48db9fa780cb41bd score=83.53 start=339.0 end=370.0 confidence=high
video_id=65d5fbad48db9fa780cb41bd score=83.41 start=122.0 end=283.0 confidence=high
video_id=65d5fbad48db9fa780cb41bd score=83.35 start=612.0 end=640.0 confidence=high

Migrate your queries

This section shows how you can convert single queries into combined queries. For example, the code that uses the single query format to find the exact moments when a red car appears or is mentioned in the conversation should look similar to the following example:

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query="red car",
    options=["visual", "conversation"],
)

const data =  {
  'query': 'red car',
  'index_id': '<YOUR_INDEX_ID>',
  'search_options': ['visual', 'conversation'],
  'operator': 'or'
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

To use the combined query format, the code should be updated as follows:

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: {
    $or: [
      {
        text: 'red car',
      },
      {
        text: 'red car',
        search_options: ['conversation'],
      },
    ],
  },
  options: ['visual'],
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence} metadata=${JSON.stringify(clip.metadata)}`,
    );
  });
}

const data =  {
  'index_id': '<YOUR_INDEX_ID>',
  'search_options': ['visual'],
  'query': {
    '$or': [
      {
        'text': 'red car'
      },
      {
        'text': 'red car',
        'search_options': ['conversation']
      }
    ]
  }
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)

const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

Pagination and filtering

Understanding and utilizing pagination and filtering enhances your ability to find and organize information. Follow the links below for more details: