Search results

When you perform a search, the API service returns all the relevant matches. Filtering narrows the scope of your query. The API service allows you to filter your search results based on metadata or the level of confidence that the search results match your query.

📘

Note

The examples on this page are specific to using simple queries. However, the principles demonstrated are similar when using combined queries.

For a description of each field in the request and response, see the API Reference > Make a search request page.

Note that, although the example code in this guide is written in Python and Node.js the API is compatible with most programming languages, and you can also use Postman or other REST clients to send requests and view responses.

Filtering search results based on metadata

To filter your search results based on metadata, add the filter parameter to the body of your request.

The filter parameter is of type Object and can contain one or more of the following fields:

  • id: An array of strings that restricts your search results based on the specified video IDs.
  • duration: A numeric field that restricts your search results based on duration, expressed in seconds.
  • width: A numeric field that restricts your search results based on width.
  • height: A numeric field that restricts your search results based on height.
  • size: A numeric field that restricts your search results based on size, expressed in bytes.

To indicate the relationship between a field and its value, you can use the exact match or comparison operators.

Exact match operator

The exact match operator matches only the results that equal the value you specify. The syntax is as follows: <field>: <value>.

Comparison operators

Use the comparison operators (lte and gte) to match based on the arithmetic comparison. The syntax is as follows: <field>:{"gte": <value>, "lte": <value}.

Filter composition

You can filter on multiple fields by adding the fields you want to filter on to the filter query parameter as shown in the Filter on size, width, and height section below.

Examples

The examples in this section assume the following:

  • A valid Twelve Labs account. For details about creating an account and retrieving your API key, see the Authentication page.
  • You’re familiar with the concepts that are described on the Quickstart page.
  • You’ve created at least one index, and the unique identifier of your index is stored in a variable named INDEX_ID. For details, see the Creating indexes page.
  • You’ve uploaded at least one video to your index, and the API service has finished indexing it. For details, see the Uploading videos page.

For a description of each field in the request and response, see the API Reference > Make a search request page.

Note that, although the example code in this guide is written in Python and Node.js the API is compatible with most programming languages, and you can also use Postman or other REST clients to send requests and view responses.

Filter on a specific video ID

The following example code uses the id field of the filter query parameter to filter on a specific video ID:

SEARCH_URL = f"{API_URL}/search/"

data = {
  "query": "bear",
  "index_id": INDEX_ID,
  "search_options": ["conversation"],
  "filter": {
    "id": ["629deb6ff05c91527bc5c1c9"]
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f"Status code: {response.status_code}")
pprint(response.json())
const SEARCH_URL = `${API_URL}/search/`

const data = JSON.stringify(
    {
    'query': 'bear',
    'index_id': INDEX_ID,
    'search_options': ['conversation'],
    'filter': {
    'id': ['629deb6ff05c91527bc5c1c9']
    }
})
const config = {
    method: 'post',
    url: SEARCH_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The following example output was truncated for brevity:

Status code: 200
{'conversation_option': 'semantic',
 'data': [{'confidence': 'low',
           'end': 433.42,
           'metadata': [{'text': "Leave it alone, I have a gun, I'm walking "
                                 "backwards on the trail back to the camp, I'm "
                                 'walking down the trail and mom and two cubs '
                                 'so I am continually to walk backwards.',
                         'type': 'conversation'}],
           'score': 72.95,
           'start': 416.65,
           'video_id': '629deb6ff05c91527bc5c1c9'},
          {'confidence': 'low',
           'end': 279.86,
           'metadata': [{'text': 'Mhm Well that puts an end to my tarpon '
                                 "fishing, get back now when you're playing "
                                 "with lions, it's important to never show "
                                 'fear chuck.',
                         'type': 'conversation'}],
           'score': 68.81,
           'start': 201.74,
           'video_id': '629deb6ff05c91527bc5c1c9'},
          {'confidence': 'low',
           'end': 490.25,
           'metadata': [{'text': 'The mom and the cubs keep following me and '
                                 'or walking at least as fast as I am. People '
                                 'get it. Oh, you, you, you, oh, oh no!',
                         'type': 'conversation'}],
           'score': 68.74,
           'start': 436.88,
           'video_id': '629deb6ff05c91527bc5c1c9'}],
 'page_info': {'limit_per_page': 10,
               'page_expired_at': '2022-06-07T04:14:51Z',
               'total_results': 5},
 'query': 'bear',
 'search_options': ['conversation'],
 'search_pool': {'index_id': '629deb409ea24f052b971993',
                 'total_count': 3,
                 'total_duration': 2198}
}

Note that this query returns only the results found in the specified videos.

Filter on multiple video IDs

The following example code uses the id field of the filter query parameter to filter on multiple video IDs:

SEARCH_URL = f"{API_URL}/search/"

data = {
  "query": "bear",
  "index_id": INDEX_ID,
  "search_options": ["conversation"],
  "filter": {
    "id": ["629deb6ff05c91527bc5c1c9" , "629deb9df05c91527bc5c1cb"]
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f"Status code: {response.status_code}")
pprint(response.json())
const SEARCH_URL = `${API_URL}/search/`

const data = JSON.stringify(
    {
    'query': 'bear',
    'index_id': INDEX_ID,
    'search_options': ['conversation'],
    'filter': {
      'id': ['629deb6ff05c91527bc5c1c9', '629deb9df05c91527bc5c1cb']
    }
})
const config = {
    method: 'post',
    url: SEARCH_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The following example output was truncated for brevity:

Status code: 200
{'conversation_option': 'semantic',
 'data': [{'confidence': 'high',
           'end': 937.75,
           'metadata': [{'text': 'Bear encounters can be scary, exciting. And '
                                 'even funny. It all really depends on the '
                                 "circumstances you're in.",
                         'type': 'conversation'}],
           'score': 88.4,
           'start': 930.44,
           'video_id': '629deb9df05c91527bc5c1cb'},
          {'confidence': 'high',
           'end': 618.76,
           'metadata': [{'text': 'bear in this next video took things a step '
                                 "further by actually breaking into a person's "
                                 'home in pursuit of none other than some '
                                 'leftover pizza. Okay.',
                         'type': 'conversation'}],
           'score': 87.82,
           'start': 605.34,
           'video_id': '629deb9df05c91527bc5c1cb'},
          {'confidence': 'high',
           'end': 805.3,
           'metadata': [{'text': ' If you were driving through the forest and '
                                 'saw a bear up ahead, what would you do? '
                                 "Surely you'd feel pretty safe knowing you "
                                 'were inside your car. But what if that bear '
                                 'came up to you and began peeking through '
                                 "your window. That's exactly the situation.",
                         'type': 'conversation'}],
           'score': 84.73,
           'start': 788.34,
           'video_id': '629deb9df05c91527bc5c1cb'}],
 'page_info': {'limit_per_page': 10,
               'next_page_token': '48b13711-21e2-4651-8890-52f26a4769ea-1',
               'page_expired_at': '2022-06-07T04:46:19Z',
               'total_results': 40},
 'query': 'bear',
 'search_options': ['conversation'],
 'search_pool': {'index_id': '629deb409ea24f052b971993',
                 'total_count': 3,
                 'total_duration': 2198}
}

Note that this query returns the results found in both videos.

Filter on duration

The example code below uses the duration field of the filter query parameter to return only the matches found in videos that are longer than or equal to 700 seconds AND shorter than or equal to 750 seconds:

SEARCH_URL = f"{API_URL}/search/"

data = {
  "query": "bear",
  "index_id": INDEX_ID,
  "search_options": ["conversation"],
  "filter": {
    "duration": {
        "gte" : 700, "lte": 750
      }
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f"Status code: {response.status_code}")
pprint(response.json())
const SEARCH_URL = `${API_URL}/search/`

const data = JSON.stringify(
    {
    'query': 'bear',
    'index_id': INDEX_ID,
    'search_options': ['conversation'],
    'filter': {
      'duration': {
        'gte' : 700, 'lte': 750
      }
    }
})
const config = {
    method: 'post',
    url: SEARCH_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The output should look similar to the following one:

Status code: 200
{'conversation_option': 'semantic',
 'data': [{'confidence': 'high',
           'end': 480.35,
           'metadata': [{'text': 'Who? A couple of big bears. Hey bear! Hey '
                                 "bear, go away bear. I don't live there. "
                                 "Don't jump at me bear. I want no trouble.",
                         'type': 'conversation'}],
           'score': 92.28,
           'start': 452.14,
           'video_id': '629deb89f05c91527bc5c1ca'},
          {'confidence': 'mid',
           'end': 521.16,
           'metadata': [{'text': "You got nothing to hump at me for they won't "
                                 'let me bring my bear mace on the plane. '
                                 "You're all good. Keep going there. Cool. "
                                 "That's a lot.",
                         'type': 'conversation'}],
           'score': 80.41,
           'start': 482.54,
           'video_id': '629deb89f05c91527bc5c1ca'}],
 'page_info': {'limit_per_page': 10,
               'page_expired_at': '2022-06-07T05:01:01Z',
               'total_results': 2},
 'query': 'bear',
 'search_options': ['conversation'],
 'search_pool': {'index_id': '629deb409ea24f052b971993',
                 'total_count': 3,
                 'total_duration': 2198}
}

Filter on size, width, and height

The example code below uses the size, width, and height fields of the filter query parameter to return only the matches found in videos that meet all the following criteria:

  • Size is greater than or equal to 50000000 bytes and less and equal to 53000000 bytes.
  • Width is greater than or equal to 850.
  • Height is greater than or equal to 400 and less and equal to 500.
SEARCH_URL = f"{API_URL}/search/"

data = {
  "query": "bear",
  "index_id": INDEX_ID,
  "search_options": ["conversation"],
  "filter": {
    "size": {
      "gte": 50000000, "lte": 53000000
    },
    "width":
      {
        "gte": 850
      },
    "height":
      {
        "gte": 400, "lte": 500
      }
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f"Status code: {response.status_code}")
pprint(response.json())
const SEARCH_URL = `${API_URL}/search/`

const data = JSON.stringify(
    {
    'query': 'bear',
    'index_id': INDEX_ID,
    'search_options': ['conversation'],
    'filter': {
      'size': {
        'gte': 50000000, 'lte': 53000000
      },
      'width':
        {
          "gte": 850
        },
      'height':
        {
          'gte': 400, 'lte': 500
        }
    }
})
const config = {
    method: 'post',
    url: SEARCH_URL,
    headers: headers,
    data: data
}
const resp = await axios(config)
const response = await resp.data
console.log(`Status code: ${resp.status}`)
console.log(response)

The following example output was truncated for brevity:

Status code: 200
{
  "conversation_option": "semantic",
  "data": [
    {
      "score": 88.4,
      "start": 930.44,
      "end": 937.75,
      "metadata": [
        {
          "type": "conversation",
          "text": "Bear encounters can be scary, exciting. And even funny. It all really depends on the circumstances you're in."
        }
      ],
      "video_id": "629deb9df05c91527bc5c1cb",
      "confidence": "high"
    },
    {
      "score": 84.74,
      "start": 937.79,
      "end": 947.56,
      "metadata": [
        {
          "type": "conversation",
          "text": "While bears aren't necessarily always aggressive creatures, it's probably best to keep your distance. Should you ever run into one in the wild?"
        }
      ],
      "video_id": "629deb9df05c91527bc5c1cb",
      "confidence": "high"
    },
    {
      "score": 84.73,
      "start": 788.34,
      "end": 805.3,
      "metadata": [
        {
          "type": "conversation",
          "text": " If you were driving through the forest and saw a bear up ahead, what would you do? Surely you'd feel pretty safe knowing you were inside your car. But what if that bear came up to you and began peeking through your window. That's exactly the situation."
        }
      ],
      "video_id": "629deb9df05c91527bc5c1cb",
      "confidence": "high"
    }
  ],
  "page_info": {
    "limit_per_page": 10,
    "next_page_token": "c1f893d1-0d05-4185-8181-394529c9dca6-1",
    "page_expired_at": "2022-06-07T05:16:05Z",
    "total_results": 35
  },
  "query": "bear",
  "search_options": [
    "conversation"
  ],
  "search_pool": {
    "index_id": "629deb409ea24f052b971993",
    "total_count": 3,
    "total_duration": 2198
  }
}

Filter on custom metadata

The example code below filters on a custom field named needs_review of type boolean. For details about specifying custom metadata, see the Update video information page. The API service will return only the matches found in videos that are tagged with needs_review: true.

headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data = {
  "query": "red car",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "filter": {
    "needs_review": True
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())
const SEARCH_URL = `${API_URL}/search`

const data = {
  "query": "bear chasing a man",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "filter": {
    "needs_review": true
  }
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following output was truncated for brevity:

Status code: 200
{
  "search_pool": {
    "total_count": 13,
    "total_duration": 8731,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "score": 84.52,
      "start": 409.95625,
      "end": 417.6375,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "63a2a2a7ce36463e0199c8e5",
      "confidence": "high",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a2a2a7ce36463e0199c8e5/410.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221228%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221228T164646Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=baf02f7a95bb2bc4d38f6b0a776e4382c4ff2a20ca9a75a021c6f4d3f385583a"
    },
    {
      "score": 66.62,
      "start": 81.64999999999999,
      "end": 83.2625,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "63a2a2a7ce36463e0199c8e5",
      "confidence": "low",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a2a2a7ce36463e0199c8e5/82.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221228%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221228T164646Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=395ffcb28f548f879c15ab3004c4e975c89a99b0b4dc2300b7bbd61af5a1f54b"
    },
    {
      "score": 64.48,
      "start": 554.21875,
      "end": 564.625,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "63a2a2a7ce36463e0199c8e5",
      "confidence": "low",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a2a2a7ce36463e0199c8e5/555.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221228%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221228T164646Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=0916e50d66158aaabb500b066cee20afc1528df8fb798ce342342358d96bd723"
    }
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 32,
    "page_expired_at": "2022-12-28T16:46:46Z",
    "next_page_token": "5f5e2da3-f8ff-4f49-a001-bba58b5d5e83-1"
  }
}

Filtering on the level of confidence

The level of confidence represents the degree of similarity between your query and the search results. You can use the threshold parameter to narrow down a response obtained from the API service and retrieve only the most relevant results. This parameter can take one of the following values:

  • low (this is the default value)
  • medium
  • high

The following example code specifies that the minimum level of confidence fragments shouldn't be lower than medium:

headers = {
  "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data = {
  "query": "car accidents",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "threshold": "medium",
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f"Status code: {response.status_code}")
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data = {
  "query": "car accidents",
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "threshold": "medium",
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following example output was truncated for brevity:

Status code: 200
{
  "search_pool": {
    "total_count": 15,
    "total_duration": 10228,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "score": 80.28,
      "start": 25.421875,
      "end": 31.375,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "63b2c707ce36463e0199c906",
      "confidence": "medium"
    },
    {
      "score": 75.25,
      "start": 266.765625,
      "end": 270.5,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium"
    {
      "score": 75.09,
      "start": 119.796875,
      "end": 125.71875,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "high"
    },
    {
      "score": 73.96,
      "start": 300.475,
      "end": 304.15625,
      "metadata": [
        {
          "type": "visual"
        }
      ],
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "high"
    }
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 4,
    "page_expired_at": "2023-01-17T08:47:23Z"
  }
}

In the example output above, note that the data array is composed of a list of objects. Each object corresponds to a matching video segment and has a field named confidence whose value is either medium or high.