Combined queries allow you to search for different elements spanning across multiple sources of information. A combined query can include any number of subqueries linked with any number of operators. Combined queries are executed in one API request and offer the following main benefits:

Enhanced flexibility: By specifying multiple conditions, you can build combined queries that, depending on your needs, are either very specific or broad.
Improved precision: Combined queries allow you to be specific about what you want to find in your videos, which can help you retrieve only the most relevant results.

For a description of each field in the request and response, see the API Reference > Make a search request page.

Prerequisites

The examples in the following sections assume the following:

You’re familiar with the concepts that are described on the Platform overview page.
You’ve uploaded at least one video to your index, and the platform has finished indexing it. For details, see the Upload videos page.

The basics of combined queries

The most important parameters that you must specify to perform a search request using combined queries are described below:

index_id: The unique identifier of the index to search.
search_options: An array of strings specifying the source of information the platform uses when performing a search. The following values are supported: visual, conversation, text_in_video, and logo. For details, see the Search options page.
(Optional) conversation_option: A string that specifies the type of search the platform will perform. The following values are supported: semantic, exact_match. For details, see the Conversation option page.
query: An object that defines a combined query. In its simplest form, the query object is composed of the following key-value pairs:
- text: A string representing your search terms.
- (Optional)search_options: If set, overrides the search options for this query.
- (Optional) conversation_option: If set, overrides the conversation option for this query.

The following example searches for car accidents based on visual cues:

SEARCH_URL = f"{API_URL}/search"

data =  {
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "query": {
    "text": "car accidents"
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data =  {
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'query': {
    'text': 'car accidents'
  }
}
const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following output was truncated for brevity:

Status code: 200
{
  "search_pool": {
    "total_count": 13,
    "total_duration": 8731,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "score": 81.79,
      "start": 117,
      "end": 125,
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/639963a1ce36463e0199c8c7/118.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221227%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221227T065557Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=8d71ebfe5d2592a1143a036bdafef6c6ae4afb199d0edb565cc37f7c9f15f018"
    },
    {
      "score": 75.61,
      "start": 420,
      "end": 430,
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/639963a1ce36463e0199c8c7/421.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221227%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221227T065558Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=4995c1fdbf5f54386feb5c87b3db0977f98bc7b1fc58ef22329a00787d51f45a"
    },
    {
      "score": 72.9,
      "start": 15,
      "end": 18,
      "video_id": "63a09373ce36463e0199c8de",
      "confidence": "low",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a09373ce36463e0199c8de/16.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221227%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221227T065557Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=938bca525893f3ae5ff76d8581276287fd9769bd7b7f6026bf385f91a8e4def8"
    },
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 137,
    "page_expired_at": "2022-12-27T06:55:58Z",
    "next_page_token": "8fef853f-1df8-406d-bae5-6899594fc825-1"
  }
}

Using operators

The platform allows you to build combined queries using the following operators:

AND: A dictionary where the key is the $and string and the value is an array of objects. Each object is a subquery. The platform returns the video clips for which all the specified queries match.
OR: A dictionary where the key is the $or string and the value is an array of objects. Each object is a subquery. The platform returns the video clips for which at least one of the queries matches.
NOT: A dictionary where the key is the $not string and the value is a dictionary composed of two queries named origin and sub. The platform returns the video clips that match the origin query but do not match the sub query. Note that the origin and sub queries can include any number of subqueries.
THEN: A dictionary where the key is the $then string, and the value is an array of objects. Each object is a subquery. The platform will return only the results for which the order of the matching video clips is the same as the order of your queries.

The following is an example of a combined query that finds the moments in your videos where a blue or a red car appears:

SEARCH_URL = f"{API_URL}/search"

data =  {
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "query": {
    "$or": [
      {
        "text": "red car"
      },
      {
        "text": "blue car"
      }
    ]
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data =  {
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'query': {
    '$or': [
      {
        'text': 'red car'
      },
      {
        'text': 'blue car'
      }
    ]
  }
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following is an example of a combined query that finds the moments in your video where:

Someone is cooking
Italian food is mentioned in the conversation
Neither the word spaghetti is displayed on the screen nor lasagna is mentioned in the conversation.

SEARCH_URL = f"{API_URL}/search"

data = {
    "index_id": INDEX_ID,
  	"search_options": ["visual"],
    "query": {
        "$not": {
            "origin": {
                "$and": [
                    {
                        "text": "Someone is cooking",
                    },
                    {
                        "text": "Italian food",
                        "search_options": ["conversation"]
                    }
                ]
            },
            "sub": {
                "$or": [
                    {
                        "text": "spaghetti",
                        "search_options": ["text_in_video"]
                    },
                    {
                        "text": "lasagna",
                        "search_options": ["conversation"]
                    }
                ]
            }
        }
    }
}
response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`;

const data = {
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'query': {
    '$not': {
      'origin': {
        '$and': [
          {
            'text': 'Someone is cooking',
            'search_options': ['visual'],
          },
          {
            'text': 'Italian food',
            'search_options': ['conversation'],
          },
        ],
      },
      'sub': {
        '$or': [
          {
            'text': 'spaghetti',
            'search_options': ['text_in_video'],
          },
          {
            'text': 'lasagna',
            'search_options': ['conversation'],
          }
        ]
      }
    }
  }
};

const resp = await axios.post(SEARCH_URL, data, {
  headers: {
    "x-api-key": API_KEY,
  },
});
const { data: response } = resp;
console.log(`Status code: ${resp.status}`);
console.log(JSON.stringify(response, null, 4));

In the example code above, note that the value of the search_options parameter is set to ["visual"] for the entire query, and it is overridden on a per-subquery basis.

Advanced techniques

This section illustrates some advanced techniques you can use to extend the functionality of combined queries.

Time-proximity search

When building combined queries, you can use the proximity parameter to extend the lower and upper boundaries of each subquery. This parameter is expressed in seconds.

Note that the proximity parameter can only be used when your combined query is composed of two or more subqueries.

The following example code uses the AND operator and the proximity parameter to find all car accidents that happened within 30 seconds before someone wins a race:

SEARCH_URL = f"{API_URL}/search"

data =  {
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "query": {
    "$and": [
      {
        "text": "winning the race"
      },
      {
        "text": "car accident"
      }
    ],
    "proximity": 30
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data =  {
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'query': {
    '$and': [
      {
        'text': 'winning the race'
      },
      {
        'text': 'car accident'
      }
    ],
    'proximity': 30
  }
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

Specifying the order of the matching video clips

The example code below uses the THEN operator to find the moments in your videos where the following occur in the specified order:

A player makes a long pass.
A player dribbles past a defender.

SEARCH_URL = f"{API_URL}/search"

data = {
    "index_id": INDEX_ID,
    "search_options": ["visual"],
    "query": {
        "$then": [
            {
                "$or": [
                    {
                        "text": "Player makes a long pass",
                    },
                    {
                        "text": "Amazing pass",
                        "search_options": ["conversation"]
                    }
                ]
            },
            {
                "text": "Player dribbles past a defender",
            }
        ]
    }
}
response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`;

const data = {
  'index_id': INDEX_ID,
  'search_options': 'visual'],
  query: {
    '$then': [
      {
        'text': 'Player is making a long pass'
      },
      {
        'text': 'Player is dribbling past a defender'
      }
    ]
  }
};

const resp = await axios.post(SEARCH_URL, data, {
  headers: {
    "x-api-key": API_KEY,
  },
});
const { data: response } = resp;
console.log(`Status code: ${resp.status}`);
console.log(JSON.stringify(response, null, 4));

The example code below combines the THEN and OR operators to find all the moments in your videos where the following occur in the specified order:

A player makes a long pass OR the words "amazing pass" are mentioned in the conversation
A player dribbles past a defender.

SEARCH_URL = f"{API_URL}/search"

data = {
    "index_id": INDEX_ID,
    "search_options": ["visual"],
    "query": {
        "$then": [
            {
                "$or": [
                    {
                        "text": "Player makes a long pass"
                    },
                    {
                        "text": "Amazing pass",
                        "search_options": ["conversation"]
                    }
                ]
            },
            {
                "text": "Player dribbles past a defender"
            }
        ]
    }
}
response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`;

const data = {
  'index_id': INDEX_ID,
  'search_options': ['visual'], 
  query: {
    '$then': [
      {
        '$or': [
          {
            'text': 'Player makes a long pass'
          },
          {
            'text': 'Amazing pass',
            'search_options': ['conversation'],
          }
        ]
      },
      {
        'text': 'Player dribbles past a defender'
      }
    ]
  }
};

const resp = await axios.post(SEARCH_URL, data, {
  headers: {
    'x-api-key': API_KEY,
  },
});
const { data: response } = resp;
console.log(`Status code: ${resp.status}`);
console.log(JSON.stringify(response, null, 4));

Migrating your queries

This section shows how you can transform single queries into new queries. For example, the code that uses the single query format to find the exact moments when a red car appears or is mentioned in the conversation should look similar to the following example:

data =  {
  "query": "red car",
  "index_id": INDEX_ID,
  "search_options": ["visual", "conversation"],
  "operator": "or"
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const data =  {
  'query': 'red car',
  'index_id': INDEX_ID,
  'search_options': ['visual', 'conversation'],
  'operator': 'or'
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    'headers': {
      'x-api-key': API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

To use the combined query format, the code should be updated as follows:

data =  {
  "index_id": INDEX_ID,
  "search_options": ["visual"],
  "query": {
    "$or": [
      {
        "text": "red car"
      },
      {
        "text": "red car",
        "search_options": ["conversation"]
      }
    ]
  }
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const data =  {
  'index_id': INDEX_ID,
  'search_options': ['visual'],
  'query': {
    '$or': [
      {
        'text': 'red car'
      },
      {
        'text': 'red car',
        'search_options': ['conversation']
      }
    ]
  }
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)

const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

Pagination

This endpoint supports pagination. For details, see the Pagination > Search results page.

Filtering

This endpoint supports filtering. For details, see the Filtering > Search results page.