Query engineering is the process of crafting and refining search queries to enhance the relevancy and accuracy of your search results. The key to successful query engineering is understanding the platform's capabilities and then formulating your queries to maximize these capabilities effectively.

Steps in query engineering

While there isn't a singular approach to querying our multimodal foundation model, the steps presented in this section will enhance your querying engineering proficiency and help you develop an understanding through experimentation.

Clearly define the objective: Determine what you want to retrieve. It can be a specific action, event, object, or something more complex, like a series of activities or people with various attributes.
Choose the type of query: Based on the objective established in the previous step, select the appropriate type of query - either single or combined. Choose a single query to search for matches of the same query across multiple sources of information. Choose a combined query to search for different elements spanning across multiple sources of information.
Create the initial query: Craft your initial query, specifying the sources of information you want to search within.
Refine and improve: If the platform isn't providing the expected search results, make your prompt more explicit. For example, instead of using broad terms like man or guitar, use more descriptive phrases like man playing guitar in the rain. Experiment with different adjectives, sequences, or sources of information. This helps in narrowing down the search.

Tips for writing better queries

Effective query engineering can significantly enhance the quality of results returned by the platform. The tips in this section will help you write better queries.

Be as specific and descriptive as possible

When crafting a query, specificity is often paramount. While single keywords might yield many results, they often don't capture the context. Thus, using descriptive phrases significantly enhances the relevance of your search results.

Examples:

Instead of dancer, use ballet dancer performing on a stage with dim lighting. The additional context gives the platform insight into the type of dance and ambiance.
Instead of dog, use dog playing fetch in a park. The revised query clearly specifies an activity and setting.

See the Find the perfect adrenaline rush activity for your next vacation section below for a practical example of how using descriptive phrases significantly improves the accuracy of your search results.

Use synonyms

Every word comes with a set of synonyms that can offer slight variances in meaning. If the platform doesn't return the expected results when using a specific word, try its synonyms.

Examples:

Instead of automobile, consider car or vehicle to narrow down your search results.
Instead of house, experiment with bungalow or villa to target a particular style.

Consider subjectivity

Videos often contain content that can be open to interpretation, given the inherently subjective nature of certain concepts. When searching for such information, it's important to understand that your query may not capture the full breadth of relevant content. Thus, iterative querying (adjusting, refining, and trying different phrasings or perspectives) becomes essential.

Examples:

Using beautiful scenery as your search query might return both a serene mountain or a beach sunset. To refine this query, specify either snow-capped mountain scenery or tropical beach at sunset.
Using dogs doing something funny as your search query might return both a dog chasing its tail or a puppy getting startled by a harmless object. Similar to the above, refine this query by specifying what you find humorous.

Use natural language

Instead of relying on traditional keyword-based search terms, help the platform deliver more accurate and contextually relevant results by expressing your queries in a more conversational or natural manner.

Examples:

Instead of guitar, solo, concert, use footage where a guitarist plays a solo during a concert.
Instead of cat, dog, chase, use videos where a cat is chasing a dog.

Rephrase domain-specific terms

Occasionally, the platform might not accurately recognize domain-specific terms. A more descriptive query that uses everyday language can enhance the accuracy of your search results in such instances.

Examples:

In a football context, instead of hurdle, use a football player jumping over another player who has their feet on the ground.
In mountaineering, instead of alpenglow, use a reddish glow near the horizon.

See the Find football hurdles section below for a practical example of how rephrasing domain-specific terms can enhance the precision of your search results.

Practical examples of query engineering

The examples in this section are from the Playground. However, the principles demonstrated are similar when invoking the API programmatically.

Find the perfect adrenaline rush activity for your next vacation

This example demonstrates how using descriptive phrases significantly enhances the accuracy of your search results.

Objective: Find a video showing the perfect adrenaline rush activity for your next vacation.

Initial query: Use adrenaline rush as your initial query. As shown in the screenshot below, the search results might be too broad:

Step-by-step refinement:

Refine your query by specifying the location - adrenaline rush activities on the beach.
In the screenshot above, the first search result is from a video showing people skydiving from a helicopter, and you want to find more videos similar to this. To do so, you can refine your query even further by specifying both the activity and the location -People skydiving from a helicopter and landing on the beach. Now, the search results are narrowed down:

Find football hurdles

This example demonstrates how using everyday language instead of jargon can improve the precision of the search results.

Objective: Find videos showing football hurdles.

Initial query: Use football hurdles as your initial query. As shown in the screenshot below, the search results are too broad, and only a few of them show the exact moments when players hurdle over their opponents:

Step-by-step refinement:

Use everyday language to refine your query - A football player jumping over another player who has their feet on the ground. Now, the search results are narrowed down:

Next steps

Depending on your use case, see the Use single queries or Use combined queries sections for step-by-step guides on using the Twelve Labs Video Understanding Platform to perform search requests.