Evaluate Agent Performance AB-620 Practice Questions (Page 2)

Questions 6–10

1. 6expert · hard

   A company is building an AI agent for a niche domain with very little real data. They need to create a test set to evaluate the agent. What is the best approach?

   • ACreate a small test set from available real data and supplement with expert-written synthetic scenarios.
   • BSkip creating a test set and evaluate the agent in production.
   • CUse a large synthetic test set generated by an LLM without expert review.
   • DUse only the available real data, even if it is very small.
   Reveal Solution Discussion Select an answer first
2. 7expert · hard

   A company is creating a test set for an AI agent that handles customer complaints. They have a limited number of real complaint queries, but they need to evaluate the agent's performance on a wide range of complaint types and tones. They also want to ensure the test set is not biased toward the most common complaints. What is the best approach?

   • ARandomly select queries from the real data without regard to complaint type or tone.
   • BUse only synthetic queries to ensure full control over the test set.
   • CUse only the real complaint queries available, since they are authentic.
   • DSupplement the real queries with synthetic queries that cover underrepresented complaint types and tones.
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   Which evaluation method is best suited for assessing an AI agent's performance when there are no predefined correct answers, such as in an open-ended creative writing task?

   • AHuman evaluation
   • BAutomated metrics like response length
   • CError rate measurement
   • DGround truth-based evaluation
   Reveal Solution Discussion Select an answer first
5. 9expert · hard

   An AI agent for a telecom company was evaluated with a test set of 1,000 queries. The results show high accuracy overall, but the agent fails on queries from users with strong accents or non-standard phrasing. What is the most likely cause and what should you do?

   • AThe evaluation method is too strict; switch to a more lenient metric.
   • BThe test set is too large; reduce it to focus on common queries.
   • CThe test set lacks diversity in language patterns; augment it with varied phrasings and re-evaluate.
   • DThe agent's temperature is too low; increase it to handle varied phrasing.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A company is creating a test set for an AI agent that handles customer complaints. They want to include a variety of complaint types and tones. What is the best way to build this test set?

   • ACollect real complaints from customer service logs and supplement with synthetic variations.
   • BUse only synthetic complaints generated by an LLM.
   • CAsk customer service agents to write sample complaints.
   • DUse only real complaints from the last month.
   Reveal Solution Discussion Select an answer first