Request a Demo

Using combined queries

Suggest Edits

📘

Note

The combined queries feature is in beta and is subject to change.

A combined query can include any number of subqueries linked with any number of operators. Combined queries are executed in one API request and offer the following main benefits:

  • Enhanced flexibility: By specifying multiple conditions, you can build combined queries that, depending on your needs, are either very specific or broad.
  • Improved precision: Combined queries allow you to be specific about what you want to find in your videos, which can help you retrieve only the most relevant results.

For a description of each field in the request and response, see the API Reference > Make a search request page.

Although the example code in this guide is written in Python and Node.js, the API is compatible with most programming languages, and you can also use Postman or other REST clients to send requests and view responses.

Prerequisites

The examples in the following sections assume the following:

  • You’re familiar with the concepts that are described on the Understand page.
  • You’ve uploaded at least one video, and the API service has finished indexing it. For details, see the Uploading videos page.

The basics of combined queries

In its simplest form, the query object is composed of the following key-value pairs:

  • text: A string representing your search terms.
  • option: A string specifying the source of information the API service uses when performing a search. The following values are supported:
    • visual: Allows you to search by objects, actions, sounds, movements, places, situational events, and complex audio-visual text descriptions.
    • conversation: Allows you to find the exact moment in your video where the specified word or phrase is mentioned.
    • text_in_video: Allows you to search for text that appears in your videos (OCR).
  • (Optional) conversation_option: A string that specifies the type of search the API service will perform. The following values are supported: semantic, transcription. For details, see the Conversation option page.

The following example searches for car accidents based on visual cues:

headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data =  {
  "query": {
    "text": "car accidents",
    "option": "visual",
    "conversation_option": "semantic"
  },
  "index_id": INDEX_ID
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`

const data =  {
  "query": {
    "text": "car accidents",
    "option": "visual",
    "conversation_option": "semantic"
  },
  "index_id": INDEX_ID
}
const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following output was truncated for brevity:

Status code: 200
{
  "search_pool": {
    "total_count": 13,
    "total_duration": 8731,
    "index_id": "639961c9e219c90227c371a2"
  },
  "data": [
    {
      "score": 81.79,
      "start": 117,
      "end": 125,
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/639963a1ce36463e0199c8c7/118.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221227%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221227T065557Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=8d71ebfe5d2592a1143a036bdafef6c6ae4afb199d0edb565cc37f7c9f15f018"
    },
    {
      "score": 75.61,
      "start": 420,
      "end": 430,
      "video_id": "639963a1ce36463e0199c8c7",
      "confidence": "medium",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/639963a1ce36463e0199c8c7/421.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221227%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221227T065558Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=4995c1fdbf5f54386feb5c87b3db0977f98bc7b1fc58ef22329a00787d51f45a"
    },
    {
      "score": 72.9,
      "start": 15,
      "end": 18,
      "video_id": "63a09373ce36463e0199c8de",
      "confidence": "low",
      "thumbnail_url": "https://project-one-thumbnail.s3.us-west-2.amazonaws.com/63a09373ce36463e0199c8de/16.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYRWJPOVHXE5SJ77T%2F20221227%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20221227T065557Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=938bca525893f3ae5ff76d8581276287fd9769bd7b7f6026bf385f91a8e4def8"
    },
  ],
  "page_info": {
    "limit_per_page": 10,
    "total_results": 137,
    "page_expired_at": "2022-12-27T06:55:58Z",
    "next_page_token": "8fef853f-1df8-406d-bae5-6899594fc825-1"
  }
}

For a description of each field in the response, see the Make a search request page.

Using operators

The API service allows you to build combined queries using the following operators:

  • AND: A dictionary where the key is the $and string and the value is an array of objects. Each object is a subquery. The API service returns the video fragments for which all the specified queries match.
  • OR: A dictionary where the key is the $or string and the value is an array of objects. Each object is a subquery. The API service returns the video fragments for which at least one of the queries matches.
  • NOT: A dictionary where the key is the $not string and the value is a dictionary composed of two queries named origin and sub. The API service will return the video fragments that match the origin query but do not match the sub query. Note that the origin and sub queries can include any number of subqueries.
  • THEN: A dictionary where the key is the $then string, and the value is an array of objects. Each object is a subquery. The API service will return only the results for which the order of the matching video fragments is the same as the order of your queries.

The following is an example of a combined query that finds the moments in your videos where a blue or a red car appears:

headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data =  {
  "query": {
    "$or": [
      {
        "text": "red car",
        "option": "visual"
      },
      {
        "text": "blue car",
        "option": "visual"
      }
    ]
  },
  "index_id": INDEX_ID
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`
const data =  {
  "query": {
    "$or": [
      {
        "text": "red car",
        "option": "visual"
      },
      {
        "text": "blue car",
        "option": "visual"
      }
    ]
  },
  "index_id": INDEX_ID
}
const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

The following is an example of a combined query that finds the moments in your video where:

  • Someone is cooking
  • Italian food is mentioned in the conversation
  • Neither the word spaghetti is displayed on the screen nor lasagna is mentioned in the conversation.
headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data = {
    "query": {
        "$not": {
            "origin": {
                "$and": [
                    {
                        "text": "Someone is cooking",
                        "option": "visual"
                    },
                    {
                        "text": "Italian food",
                        "option": "conversation"
                    }
                ]
            },
            "sub": {
                "$or": [
                    {
                        "text": "spaghetti",
                        "option": "text_in_video"
                    },
                    {
                        "text": "lasagna",
                        "option": "conversation"
                    }
                ]
            }
        }
    },
    "index_id": INDEX_ID,
}
response = requests.post(SEARCH_URL, headers=headers, json=data)
print(f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`;

const data = {
  query: {
    $not: {
      origin: {
        $and: [
          {
            text: "Someone is cooking",
            option: "visual",
          },
          {
            text: "Italian food",
            option: "conversation",
          },
        ],
      },
      sub: {
        $or: [
          {
            text: "spaghetti",
            option: "text_in_video",
          },
          {
            text: "lasagna",
            option: "conversation",
          },
        ],
      },
    },
  },
  index_id: INDEX_ID,
};

const resp = await axios.post(SEARCH_URL, data, {
  headers: {
    "x-api-key": API_KEY,
  },
});
const { data: response } = resp;
console.log(`Status code: ${resp.status}`);
console.log(JSON.stringify(response, null, 4));

Advanced techniques

This section illustrates some advanced techniques you can use to extend the functionality of combined queries.

Time-proximity search

When building combined queries, you can use the proximity parameter to extend the lower and upper boundaries of each subquery. This parameter is expressed in seconds.

Note that the proximity parameter can only be used when your combined query is composed of two or more subqueries.

The following example code uses the AND operator and the proximity parameter to find all car accidents that happened within 30 seconds before someone wins a race:

headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data =  {
  "query": {
    "$and": [
      {
        "text": "winning the race",
        "option": "visual"
      },
      {
        "text": "car accident",
        "option": "visual"
      }
    ],
    "proximity": 30
  },
  "index_id": INDEX_ID
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`
const data =  {
  "query": {
    "$and": [
      {
        "text": "winning the race",
        "option": "visual",
      },
      {
        "text": "car accident",
        "option": "visual"
      }
    ],
    "proximity": 30
  },
  
  "index_id": INDEX_ID,
}
const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

Specifying the order of the matching video fragments

The example code below uses the THEN operator to find the moments in your videos where the following occur in the specified order:

  1. A player makes a long pass.
  2. A player dribbles past a defender.
headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data = {
    "query": {
      "$then": [
        {
          "text": "Player makes a long pass",
          "option": "visual",
        },
        {
          "text": "Player dribbles past a defender",
          "option": "visual",
        }
      ]
    },
    "index_id": INDEX_ID,
  };
response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())


const SEARCH_URL = `${API_URL}/search`;

const data = {
  query: {
    "$then": [
      {
        "text": "Player is making a long pass",
        "option": "visual",
      },
      {
        "text": "Player is dribbling past a defender",
        "option": "visual",
      }
    ]
  },
  "index_id": INDEX_ID,
};

const resp = await axios.post(SEARCH_URL, data, {
  headers: {
    "x-api-key": API_KEY,
  },
});
const { data: response } = resp;
console.log(`Status code: ${resp.status}`);
console.log(JSON.stringify(response, null, 4));

The example code below combines the THEN and OR operators to find all the moments in your videos where the following occur in the specified order:

  1. A player makes a long pass OR the words "amazing pass" are mentioned in the conversation
  2. A player dribbles past a defender.
headers = {
    "x-api-key": API_KEY
}

SEARCH_URL = f"{API_URL}/search"

data = {
  "query": {
  "$then": [
    {
      "$or": [
        {
          "text": "Player makes a long pass",
          "option": "visual"
        },
        {
          "text": "Amazing pass",
          "option": "conversation"
        }
      ]
    },
    {
      "text": "Player dribbles past a defender",
      "option": "visual"
    }
  ]
},
    "index_id": INDEX_ID,
  };
response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const SEARCH_URL = `${API_URL}/search`;

const data = {
  query: {
    "$then": [
      {
        "$or": [
          {
            "text": "Player makes a long pass",
            "option": "visual",
          },
          {
            "text": "Amazing pass",
            "option": "conversation",
          }
        ]
      },
      {
        "text": "Player dribbles past a defender",
        "option": "visual",
      },
    ],
  },
  index_id: INDEX_ID,
};

const resp = await axios.post(SEARCH_URL, data, {
  headers: {
    "x-api-key": API_KEY,
  },
});
const { data: response } = resp;
console.log(`Status code: ${resp.status}`);
console.log(JSON.stringify(response, null, 4));

Migrating your queries

This section shows how you can convert your query to the new format. For example, the code that uses the old format to find the exact moments when a red car appears or is mentioned in the conversation should look similar to the following example:

data =  {
  "query": "red car",
  "index_id": INDEX_ID,
  "search_options": ["visual", "conversation"],
  "operator": "or"
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const data =  {
  "query": "red car",
  "index_id": INDEX_ID,
  "search_options": ["visual", "conversation"],
  "operator": "or"
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)
const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

To use the new format, the code should be updated as follows:

data =  {
  "query": {
    "$or": [
      {
        "text": "red car",
        "option": "visual"
      },
      {
        "text": "red car",
        "option": "conversation"
      }
    ]
  },
  "index_id": INDEX_ID
}

response = requests.post(SEARCH_URL, headers=headers, json=data)
print (f'Status code: {response.status_code}')
pprint(response.json())

const data =  {
  "query": {
    "$or": [
      {
        "text": "red car",
        "option": "visual"
      },
      {
        "text": "red car",
        "option": "conversation"
      }
    ]
  },
  "index_id": INDEX_ID
}

const resp = await axios.post(
  SEARCH_URL,
  data,
  {
    "headers": {
      "x-api-key": API_KEY
    }
  }
)

const { data: response } = resp;
console.log(`Status code: ${resp.status}`)
console.log(JSON.stringify(response,null,4))

Pagination

This endpoint supports pagination. For details, see the Pagination > Search results page.

Filtering

This endpoint supports filtering. For details, see the Filtering > Search results page.

Related topics

Updated 5 days ago