After your videos have been indexed by the Pegasus video language model, you can prompt the platform to generate text based on their content. Prompt engineering is the process of iteratively refining how you craft your instructions or questions to the model to improve the quality, relevance, and precision of the responses. Prompt engineering is important for enhancing the effectiveness of the model in various use cases, from content creation and summarization to question-answering, as shown below:

Improves accuracy: Tailored prompts produce more precise outputs by clearly specifying the task.
Reduces ambiguity: Well-designed prompts limit the model's scope for interpretation, ensuring relevant responses.
Enhances efficiency: Effective prompts reduce the need for post-processing, saving time and resources.
Customizes outputs: Through prompt engineering, you can tailor the outputs to specific tones, styles, or formats, meeting diverse requirements.
Provides context: Prompts can provide essential context to the model, ensuring the output is relevant and appropriate for the given situation or domain.

Steps in prompt engineering

The typical steps involved in prompt engineering are as follows:

Define the objective: Identify what you need from the model, such as generating a description of a video segment or answering a question based on video content.
Craft the initial prompt: Based on your objective, develop the initial version of your prompt. Include all necessary details and context, and specify the expected output format.
Test and iterate: Analyze the output and refine your prompt based on the results. This step may involve several iterations.

Tips for writing better prompts

Crafting the perfect prompt is not achieved through a universal solution, as the effectiveness of a specific method can vary widely depending on the task at hand. However, the tips provided in this section can help enhance your prompt-writing skills. By experimenting with them, you can discover approaches that lead to more accurate and relevant responses.

Provide examples

Examples guide the model in generating the expected output, reducing ambiguity, and ensuring the platform generates relevant responses. The following example creates a police report based on surveillance footage. It includes an example of a similar report to guide the model's response.

Provide context

Providing context in prompts helps the platform understand your requirements, ensuring the generated response is accurately tailored to your needs. By providing context, you reduce the chances of irrelevant outputs. The following example provides the required context to customize the generated response according to your needs.

To see this example in the Playground, ensure you're logged in, and then open this URL in your browser.

Be specific

Specificity guides the model in producing highly relevant and targeted responses by aligning the output with your intentions. The following example indicates the exact aspect of the video you want the model to focus on - creating a daily workout plan for this week based on the workout routine mentioned in this video. This helps the model understand the scope of the prompt and generate a targeted response.

To see this example in the Playground, ensure you're logged in, and then open this URL in your browser.

Choose the type of prompt

Based on your requirements, differentiate between question-answering and description-based prompts, as each will guide the model's focus differently. The example prompt below is phrased as a question and instructs the model to list the filming techniques used in a video.

To see this example in the Playground, ensure you're logged in, and then open this URL in your browser.

Specify the desired style and format for the output

Clearly state the desired output’s length, style, and format (examples: JSON format, email) to ensure the output meets your requirements. The example below summarizes a video as an email, focusing on the five most important points.

To see this example in the Playground, ensure you're logged in, and then open this URL in your browser.

Choose the language for the output

Specify if you want the output in a different language. The following example summarizes a video, indicating that the response should be in Spanish.

To see this example in the Playground, ensure you're logged in, and then open this URL in your browser.

Be concise

Being concise helps the model focus on the essential information. This speeds up processing and increases the likelihood of generating precise, relevant responses.

Tune the temperature

Tuning the temperature controls the randomness of the text output. A lower temperature results in more deterministic results, which is ideal for tasks requiring high accuracy and specificity. In contrast, a higher temperature produces more creative text, which is suitable for brainstorming or creative writing tasks. Experiment with this setting to find the optimal balance that meets your objectives. For details, see the Tune the temperature page.

Practical examples of prompt engineering

The examples in this section are from the Playground. However, the principles demonstrated are similar when invoking the API programmatically.

How to cook macaroni and cheese

Objective: Write a recipe for cooking macaroni and cheese based on a video.

Initial prompt: Use "Describe the recipe shown in this video." as the initial prompt. This yields a basic summary without detailed steps or quantities for each ingredient.

Step-by-step refinement:

Modify the prompt to specify the need for more detail - "List all ingredients and steps mentioned in the video for cooking macaroni and cheese." The response now contains a detailed list of ingredients and a step-by-step process. Still, it needs to capture all the nuances of the cooking technique and measurements.
Further, refine your prompt to include cooking techniques, measurements, and any tips - "List all ingredients with measurements and detailed steps for cooking macaroni and cheese as shown in the video. Include any techniques or tips to ensure the dish turns out perfectly." This prompt captures the basic recipe and the specific nuances and precise measurements that make the dish special.

Write a workout plan

Objective: Write an email containing a workout plan based on a video.

Initial prompt: Use "List the workouts mentioned in this video." as the initial prompt. The screenshot below shows that the initial response doesn't provide details about the workouts, and it's not suitable to be used in an email:

Step-by-step refinement:

Refine your prompt by specifying that the model should briefly describe each workout - "List the workouts mentioned in this video. Add a brief description of up to two short sentences for each workout."
The screenshot below shows that the refined prompt produces a more targeted and helpful response:
Further, refine your prompt by specifying the format - "List the workouts mentioned in this video. Add a brief description of up to two short sentences for each workout. Format your response as an email." Now, the response should look like this :

To see this example in the Playground, ensure you're logged in, and then open this URL in your browser.