What does 'answer faithfulness' mean in the context of RAG evaluation?

The answer is consistent with the retrieved context

The answer uses complex vocabulary

The answer is identical to the query

The answer is generated quickly

A legal research company has built a RAG system that answers questions based on a corpus of court documents. They want to evaluate whether the system's answers are faithful to the retrieved documents and whether the retrieval returns relevant passages. Which combination of metrics should they use?

Retrieval precision and answer faithfulness (e.g., BERTScore or human review)

ROUGE and BLEU on the final answers

How does BERTScore evaluate text generation?

By using contextual embeddings to measure semantic similarity

By having humans rate the output

A company is choosing between two foundation models for a document summarization tool. Model A produces slightly higher quality summaries but has a higher cost per interaction. Model B is cheaper but produces lower quality summaries, leading to more user escalations. The company wants to maximize user satisfaction while controlling costs. Which evaluation approach should they use?

Conduct an A/B test measuring user satisfaction and cost per interaction, then calculate the cost per satisfied user

Use ROUGE to compare summary quality and choose the higher score

Choose the model with the lowest cost per interaction

Choose the model with the highest user satisfaction

AWSCertified AI Practitioner

Domain 3Objective 4

Task Statement 3.4: Describe Methods to Evaluate FM Performance. AIF-C01 Practice Questions (Page 1)

Part of the Content Domain 3: Applications of Foundation Models domain, which makes up ~32% of our current practice bank. AWS does not publish an official question count, but from its 90-minute exam (~35–60 total, ~11–19 in this domain), expect 3–5 from this objective — we provide 40 practice questions to prepare you well beyond it. (estimate)

40questions here

8free pages

14concepts

14 Concepts in This Objective

Human-in-the-loop evaluation
Benchmark datasets
Amazon Bedrock Model Evaluation
ROUGE metric
BLEU metric
BERTScore metric
LLM-as-a-judge
Business objective alignment
RAG evaluation
Agent evaluation
Workflow evaluation
Task completion rate
User satisfaction
Cost per interaction

Questions 1–5

1foundation · easy
What does 'answer faithfulness' mean in the context of RAG evaluation?
Select an answer first
2foundation · easy
Which benchmark dataset is commonly used to evaluate text generation quality in tasks like summarization?
Select an answer first
3application · medium
A legal research company has built a RAG system that answers questions based on a corpus of court documents. They want to evaluate whether the system's answers are faithful to the retrieved documents and whether the retrieval returns relevant passages. Which combination of metrics should they use?
Select an answer first
4foundation · easy
How does BERTScore evaluate text generation?
Select an answer first
5expert · hard
A company is choosing between two foundation models for a document summarization tool. Model A produces slightly higher quality summaries but has a higher cost per interaction. Model B is cheaper but produces lower quality summaries, leading to more user escalations. The company wants to maximize user satisfaction while controlling costs. Which evaluation approach should they use?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to AIF-C01

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by AWS. “AIF-C01” is a trademark of its owner, used for identification only.

Task Statement 3.4: Describe Methods to Evaluate FM Performance. AIF-C01 Practice Questions (Page 1)

Questions 1–5

What does 'answer faithfulness' mean in the context of RAG evaluation?

Which benchmark dataset is commonly used to evaluate text generation quality in tasks like summarization?

A legal research company has built a RAG system that answers questions based on a corpus of court documents. They want to evaluate whether the system's answers are faithful to the retrieved documents and whether the retrieval returns relevant passages. Which combination of metrics should they use?

How does BERTScore evaluate text generation?

Keep studying — Content Domain 3: Applications of Foundation Models