After running an evaluation, an AI agent builder sees that the agent scores 95% accuracy overall, but the test set contains 80% simple queries and only 20% complex multi-step queries. The agent fails on most of the complex ones. What is the most important insight from this result?

The overall accuracy is misleading because the test set is imbalanced; the agent performs poorly on complex queries, which need more attention.

The accuracy metric is not useful for this agent and should be replaced with a different metric.

The agent is performing well overall and no changes are needed.

The test set should be discarded because it does not reflect the real distribution of queries.

A company has an AI agent that provides technical support. They want to evaluate the agent's responses for helpfulness and safety. They have a test set of 200 queries. Which evaluation method is most appropriate?

Manual evaluation by human reviewers using a rubric for helpfulness and safety.

Automated evaluation using a simple keyword match for safety.

LLM-as-a-judge evaluation without human review.

After running an evaluation on an AI agent, you notice that the overall accuracy is 90%, but the agent fails on all queries that include the word 'refund'. What should you do first?

Ignore the failures because overall accuracy is high.

Examine the test set to see if 'refund' queries are underrepresented or have ambiguous ground truth.

Retrain the agent on the entire test set to improve accuracy.

Increase the temperature of the agent's responses to get more varied outputs.

What is the primary purpose of creating a test set when evaluating an AI agent's performance?

To train the agent's underlying language model with new examples

To store the agent's conversation logs for compliance auditing

To provide a fixed set of inputs and expected outputs used to measure the agent's behavior consistently

To replace the need for human oversight during agent deployment

Microsoft Certified:AI Agent Builder Associate

Domain 3Objective 1

Evaluate Agent Performance AB-620 Practice Questions (Page 1)

Part of the Test and manage agents domain, which accounts for 20–25% of the AB-620 exam. Microsoft does not publish an official question count, but from its 120-minute exam (~50–80 total, ~10–20 in this domain), expect 5–10 from this objective — we provide 38 practice questions to prepare you well beyond it. (estimate)

38questions here

8free pages

3concepts

20–25%of the exam

Questions 1–5

1application · medium
After running an evaluation, an AI agent builder sees that the agent scores 95% accuracy overall, but the test set contains 80% simple queries and only 20% complex multi-step queries. The agent fails on most of the complex ones. What is the most important insight from this result?
Select an answer first
2application · medium
A company has an AI agent that provides technical support. They want to evaluate the agent's responses for helpfulness and safety. They have a test set of 200 queries. Which evaluation method is most appropriate?
Select an answer first
3application · medium
After running an evaluation on an AI agent, you notice that the overall accuracy is 90%, but the agent fails on all queries that include the word 'refund'. What should you do first?
Select an answer first
4foundation · easy
What is the primary purpose of creating a test set when evaluating an AI agent's performance?
Select an answer first
5foundation · easy
When evaluating an AI agent that answers customer questions from a knowledge base, which evaluation method is most appropriate to measure whether the answers are factually correct?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to AB-620

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by Microsoft. “AB-620” is a trademark of its owner, used for identification only.