You can sort your search results using the sort_option and group_by parameters. Note that, for each matching video clip, the platform returns a field named confidence representing the level of confidence that a specific video clip matches your query.

When you don't specify the group_by parameter, the platform returns your search results as a flat list, sorted on the confidence field in descending order.

When you set the group_by parameter to video, the platform returns your search results grouped by video, sorts the clips belonging to each video on the confidence field in descending order, and you can use the sort_option parameter as follows:

Set it to clip_count to specify that the platform must determine the number of matching video clips in each video, and sort the list of videos on that value in descending order.
Set it to score to specify that the platform must determine the maximum value of the confidence field for each video, and sort the list of videos on that value in descending order.

Note that the platform doesn't support grouping when using combined queries.

For a description of each field in the request and response, see the API Reference > Make a search request page.

You can interact with the platform using one of the available SDKs or an HTTP client like requests or axios. Follow the steps in one of the sections below, depending on your use case.

Prerequisites

You're familiar with the concepts that are described on the Platform overview page.
You've already created an index, and the Marengo video understanding engine is enabled for this index.
You've already uploaded a video, and the platform has finished indexing it.

Use an SDK

Refer to this section if you use one of the available SDKs.

Sorting video clips

The following example code illustrates the default behavior of the platform. The search results are sorted on the level of confidence in descending order.

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
  index_id="YOUR_INDEX_ID", 
  query="<YOUR_QUERY>", 
  options=["visual"],
  group_by="clip", # This parameter is optional 
  sort_option="score" # This parameter is optional 
)

# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f"video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence} metadata={clip.metadata}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, SearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>'
  query: '<YOUR_QUERY>',
  options: ['visual'],
  groupBy: 'clip', // This parameter is optional
  sortOption: 'score', // This parameter is optional
});
printPage(searchResults.data);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData as SearchData[]).forEach((clip) => {
    console.log(
      `video_id= ${clip.videoId} score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence} metadata=${JSON.stringify(clip.metadata)}`,
    );
  });
}

The following output was truncated for brevity:

video_id=65e013b448db9fa780cb4379 score=56.75 start=366.4375 end=372.140625 confidence=low metadata=[{'type': 'visual'}]
video_id=65e013b448db9fa780cb4379 score=56.53 start=377.46875 end=382.25 confidence=low metadata=[{'type': 'visual'}]
video_id=65e013b448db9fa780cb4379 score=54.1 start=17.71875 end=22.875 confidence=low metadata=[{'type': 'visual'}]

In this example, note that the data array contains a list of objects, and each object corresponds to a video clip that matched your query. The data array is sorted on the value of the score field in descending order.

Sorting videos

When your search results are grouped by vides, you can use the sort_option to sort the list of videos in the response.

Sorting videos on the number of matches

The following example code groups the response by video and sorts the list of videos on the number of matches:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>, 
  query="<YOUR_QUERY>", 
  options=["visual"],
  group_by="video",
  sort_option="clip_count"
)

# Utility function to print a specific page
def print_page(page):
    for video in page:
        print(f"Video id: {video.id}")
        for clip in video.clips:
            print(
                f"video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence} metadata={clip.metadata}"
            )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, GroupByVideoSearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: '<YOUR_QUERY>',
  options: ['visual'],
  groupBy: 'video',
  sortOption: 'clip_count',
});
printPage(searchResults);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData.data as GroupByVideoSearchData[]).forEach((video) => {
    console.log(`videoId=${video.id}`);
    video.clips?.forEach((clip) => {
      console.log(
        `  score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
      );
    });
  });
}

The following output was truncated for brevity:

video_id=65e013b448db9fa780cb4379 score=85.21 start=253.484375 end=270.71875 confidence=high metadata=[{'type': 'visual'}]
video_id=65e013b448db9fa780cb4379 score=85.18 start=438.0625 end=447.09375 confidence=high metadata=[{'type': 'visual'}]
video_id=65e013b448db9fa780cb4379 score=85.11 start=541.96875 end=578.34375 confidence=high metadata=[{'type': 'visual'}]

Note the following about this example:

The data array contains a list of objects. Each object corresponds to a video and is composed of the following key-value pairs:
- id: The unique identifier of the video to which the items in the clips array belong.
- clips: A list of the video clips that match your search terms.
The list of videos is sorted on the number of video clips in each video in descending order.
The video clips in the clips arrays are sorted on the confidence field in descending order.

Sorting videos on the level of confidence

The following example code groups the response by video and sorts the list of videos on the maximum value of the confidence field:

from twelvelabs import TwelveLabs

client = TwelveLabs(api_key="<YOUR_API_KEY>")

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>", 
  query="<YOUR_QUERY", 
  options=["visual"],
  group_by="video",
  sort_option="score"
)

# Utility function to print a specific page
def print_page(page):
    for video in page:
        print(f"Video id: {video.id}")
        for clip in video.clips:
            print(
                f"video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence} metadata={clip.metadata}"
            )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

import { TwelveLabs, GroupByVideoSearchData } from 'twelvelabs-js';

const client = new TwelveLabs({ apiKey: '<YOUR_API_KEY>'});

let searchResults = await client.search.query({
  indexId: '<YOUR_INDEX_ID>',
  query: '<YOUR_QUERY>',
  options: ['visual'],
  groupBy: 'video',
  sortOption: 'score',
});
printPage(searchResults);
while (true) {
  const page = await searchResults.next();
  if (page === null) break;
  else printPage(page);
}
// Utility function to print a specific page
function printPage(searchData) {
  (searchData.data as GroupByVideoSearchData[]).forEach((video) => {
    console.log(`videoId=${video.id}`);
    video.clips?.forEach((clip) => {
      console.log(
        `  score=${clip.score} start=${clip.start} end=${clip.end} confidence=${clip.confidence}`,
      );
    });
  });
}

The following output was truncated for brevity:

Video id: 65d6131c48db9fa780cb415f
video_id=65d6131c48db9fa780cb415f score=83.3 start=273.96875 end=289.0625 confidence=high metadata=[{'type': 'visual'}]
video_id=65d6131c48db9fa780cb415f score=83.21 start=357.40625 end=370.78125 confidence=high metadata=[{'type': 'visual'}]
video_id=65d6131c48db9fa780cb415f score=83.18 start=388.0234375 end=439.84375 confidence=high metadata=[{'type': 'visual'}]

Note the following about this example:

The data array contains a list of objects. Each object corresponds to a video and is composed of the following key-value pairs:
- id: The unique identifier of a video to which the items in the clips array belong.
- clips: A list of the video clips that match your search terms.
The list of videos is sorted on the maximum value of the confidence field in descending order.
The video clips in the clips arrays are sorted on the confidence field in descending order.

Use an HTTP client

Refer to this section if you use clients such as requests or axios.

Sorting video clips

The following example code illustrates the default behavior of the platform. The search results are sorted on the level of confidence in descending order.

SEARCH_URL = f"{API_URL}/search"

data = {
  "query": "car accidents",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "group_by": "clip", # This parameter is optional 
  "sort_option": "score" # This parameter is optional
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data = {
  'query': 'car accidents',
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'group_by': 'clip', // This parameter is optional
  'sort_option': 'score' // This parameter is optional
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following output was truncated for brevity:

Status code: 200
{
  "search_pool": {
    "total_count": 15,
    "total_duration": 10228,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "score": 81.79,
      "start": 117.8359375,
      "end": 125.71875,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium"
    },

    {
      "score": 76.8,
      "start": 265.03125,
      "end": 270.5,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium"
    },
    {
      "score": 73.74,
      "start": 196.6875,
      "end": 205.78125,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "63b2c707ce36463e0199c906",
      "confidence": "low"
    }
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 229,
    "page_expired_at": "2023-01-11T08:33:46Z",
    "next_page_token": "02feedbf-3895-4d5e-b1b3-87497cf1a39c-1"
  }
}

In the example output above, note that the data array contains a list of objects, and each object corresponds to a video clip that matched your query. The data array is sorted on the value of the score field in descending order.

Sorting videos

When your search results are grouped by vides, you can use the sort_option to sort the list of videos in the response.

Sorting videos on the number of matches

The following example code groups the response by video and sorts the list of videos on the number of matches:

SEARCH_URL = f"{API_URL}/search"

data = {
  "query": "car accidents",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "group_by": "video",
  "sort_option": "clip_count"
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data = {
  'query': 'car accidents',
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'group_by': 'video',
  'sort_option': 'clip_count'
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following output was truncated for brevity:

{
  "search_pool": {
    "total_count": 15,
    "total_duration": 10228,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "clips": [
        {
          "score": 86.69,
          "start": 137.1875,
          "end": 149.1875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996260ce36463e0199c8c5",
          "confidence": "high"
        },
        {
          "score": 85.82,
          "start": 190.2125,
          "end": 193.5625,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996260ce36463e0199c8c5",
          "confidence": "high"
        },
        {
          "score": 48.69,
          "start": 7.96875,
          "end": 11.5,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996260ce36463e0199c8c5",
          "confidence": "low"
        },
        {
          "score": 47.19,
          "start": 752.609375,
          "end": 756.609375,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996260ce36463e0199c8c5",
          "confidence": "low"
        }
      ],
      "id": "63996260ce36463e0199c8c5"
    },
    {
      "clips": [
        {
          "score": 86.14,
          "start": 624.46875,
          "end": 630.46875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996246ce36463e0199c8c4",
          "confidence": "high"
        },
        {
          "score": 85.37,
          "start": 432.46875,
          "end": 442.46875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996246ce36463e0199c8c4",
          "confidence": "high"
        },
        {
          "score": 46.16,
          "start": 293.075,
          "end": 296.8125,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63996246ce36463e0199c8c4",
          "confidence": "low"
        }
      ],
      "id": "63996246ce36463e0199c8c4"
    },
    {
      "clips": [
        {
          "score": 83.97,
          "start": 409.95625,
          "end": 417.6375,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63b53433ce36463e0199c909",
          "confidence": "high"
        },
        {
          "score": 52.6,
          "start": 562.94375,
          "end": 568.1328125,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63b53433ce36463e0199c909",
          "confidence": "low"
        },
        {
          "score": 46.94,
          "start": 312.9625,
          "end": 316.825,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63b53433ce36463e0199c909",
          "confidence": "low"
        }
      ],
      "id": "63b53433ce36463e0199c909"
    }
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 5,
    "total_inner_matches": 256,
    "page_expired_at": "2023-01-09T12:32:38Z"
  }
}

Note the following about the example output above:

The data array contains a list of objects. Each object corresponds to a video and is composed of the following key-value pairs:
- id: The unique identifier of the video to which the items in the clips array belong.
- clips: A list of the video clips that match your search terms.
The list of videos is sorted on the number of video clips in each video in descending order.
The video clips in the clips arrays are sorted on the confidence field in descending order.

Sorting videos on the level of confidence

The following example code groups the response by video and sorts the list of videos on the maximum value of the confidence field:

SEARCH_URL = f"{API_URL}/search"

data = {
  "query": "car accidents",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "group_by": "video",
  "sort_option": "score"
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data = {
  'query': 'car accidents',
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'group_by': 'video',
  'sort_option': 'score'
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following output was truncated for brevity:

{
  "search_pool": {
    "total_count": 15,
    "total_duration": 10228,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "clips": [
        {
          "score": 81.79,
          "start": 117.8359375,
          "end": 125.71875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "639963a1ce36463e0199c8c7",
          "confidence": "medium"
        },
        {
          "score": 53.26,
          "start": 203.63125,
          "end": 207.1875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "639963a1ce36463e0199c8c7",
          "confidence": "low"
        },
        {
          "score": 46.69,
          "start": 483.53125,
          "end": 485.53125,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "639963a1ce36463e0199c8c7",
          "confidence": "low"
        }
      ],
      "id": "639963a1ce36463e0199c8c7"
    },
    {
      "clips": [
        {
          "score": 77.29,
          "start": 25.421875,
          "end": 31.375,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63b2c707ce36463e0199c906",
          "confidence": "medium"
        },
        {
          "score": 60.67,
          "start": 11.890625,
          "end": 25.421875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63b2c707ce36463e0199c906",
          "confidence": "low"
        },
        {
          "score": 46.16,
          "start": 619.59375,
          "end": 623.1875,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "63b2c707ce36463e0199c906",
          "confidence": "low"
        }
      ],
      "id": "63b2c707ce36463e0199c906"
    },
    {
      "clips": [
        {
          "score": 46.29,
          "start": 403.171875,
          "end": 406.25,
          "metadata": [
            {
              "type": "visual"
            }
          ],
          "video_id": "639963afce36463e0199c8c8",
          "confidence": "low"
        }
      ],
      "id": "639963afce36463e0199c8c8"
    }
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 9,
    "total_inner_matches": 229,
    "page_expired_at": "2023-01-11T12:49:15Z"
  }
}

Note the following about the example output above:

The data array contains a list of objects. Each object corresponds to a video and is composed of the following key-value pairs:
- id: The unique identifier of a video to which the items in the clips array belong.
- clips: A list of the video clips that match your search terms.
The list of videos is sorted on the maximum value of the confidence field in descending order.
The video clips in the clips arrays are sorted on the confidence field in descending order.