A startup is evaluating two foundation models for a creative writing assistant. They have a large set of prompts and need to compare the quality of the outputs. They want to automate the evaluation to reduce manual effort, but they also need to ensure the evaluation is reliable. Which approach should they use?

Use BLEU to measure translation quality

Use ROUGE to compare outputs to reference texts

Use a strong LLM as a judge to score outputs on criteria like coherence and creativity

Use task completion rate on a small sample

What does a high user satisfaction score indicate about an AI application?

Users are generally pleased with their experience

The application has low latency

The application is cost-effective

The model is technically superior

A retail company is evaluating two foundation models for their product description generator. They want to choose the model that best meets their business goal of increasing user engagement on product pages. They have data on click-through rates and time spent on pages. Which evaluation approach should they use?

Use BLEU to measure translation quality

Use ROUGE to compare descriptions to existing ones

Run an A/B test measuring click-through rates and time on page

A company has deployed an AI agent that handles customer refunds. The agent must correctly use the refund tool and complete the refund process. They have observed that the agent sometimes calls the tool with incorrect parameters, leading to failed refunds. They want to evaluate the agent's performance to identify these issues. Which evaluation approach should they use?

Use ROUGE to evaluate the agent's responses

Measure task success rate and separately track tool call correctness (e.g., parameter accuracy)

Use user satisfaction surveys only

AWSCertified AI Practitioner

Domain 3Objective 4

Task Statement 3.4: Describe Methods to Evaluate FM Performance. AIF-C01 Practice Questions (Page 8)

Part of the Content Domain 3: Applications of Foundation Models domain, which makes up ~32% of our current practice bank. AWS does not publish an official question count, but from its 90-minute exam (~35–60 total, ~11–19 in this domain), expect 3–5 from this objective — we provide 40 practice questions to prepare you well beyond it. (estimate)

40questions here

8free pages

14concepts

14 Concepts in This Objective

Human-in-the-loop evaluation
Benchmark datasets
Amazon Bedrock Model Evaluation
ROUGE metric
BLEU metric
BERTScore metric
LLM-as-a-judge
Business objective alignment
RAG evaluation
Agent evaluation
Workflow evaluation
Task completion rate
User satisfaction
Cost per interaction

Questions 36–40

36foundation · easy
Which metric is specifically designed to evaluate machine translation by measuring n-gram precision?
Select an answer first
37application · medium
A startup is evaluating two foundation models for a creative writing assistant. They have a large set of prompts and need to compare the quality of the outputs. They want to automate the evaluation to reduce manual effort, but they also need to ensure the evaluation is reliable. Which approach should they use?
Select an answer first
38foundation · easy
What does a high user satisfaction score indicate about an AI application?
Select an answer first
39application · medium
A retail company is evaluating two foundation models for their product description generator. They want to choose the model that best meets their business goal of increasing user engagement on product pages. They have data on click-through rates and time spent on pages. Which evaluation approach should they use?
Select an answer first
40expert · hard
A company has deployed an AI agent that handles customer refunds. The agent must correctly use the refund tool and complete the refund process. They have observed that the agent sometimes calls the tool with incorrect parameters, leading to failed refunds. They want to evaluate the agent's performance to identify these issues. Which evaluation approach should they use?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

No more pagesBack to AIF-C01

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by AWS. “AIF-C01” is a trademark of its owner, used for identification only.

Task Statement 3.4: Describe Methods to Evaluate FM Performance. AIF-C01 Practice Questions (Page 8)

Questions 36–40

Which metric is specifically designed to evaluate machine translation by measuring n-gram precision?

What does a high user satisfaction score indicate about an AI application?

Keep studying — Content Domain 3: Applications of Foundation Models