Evaluate Agent Performance AB-620 Practice Questions (Page 6)

Questions 26–30

1. 26expert · hard

   A team is creating a test set for an AI agent that handles insurance claims. They have a limited budget and need to evaluate the agent's performance on both common and rare claim scenarios. They also need to ensure the evaluation is repeatable and can be used to compare future versions of the agent. What is the best strategy?

   • AUse only rare scenarios in the test set, since they are the most challenging.
   • BCreate a balanced test set with a fixed set of queries covering common and rare scenarios, and use it for all future evaluations.
   • CUse a test set that changes over time to reflect evolving user behavior.
   • DCreate a new random test set for each evaluation to avoid overfitting.
   Reveal Solution Discussion Select an answer first
2. 27application · medium

   A retail company has an AI agent that recommends products. They have a large dataset of 10,000 past customer interactions with known purchase outcomes. They need to evaluate the agent's recommendation quality efficiently. Which evaluation method should they choose?

   • ALLM-as-a-judge evaluation on the full dataset without a custom metric.
   • BManual evaluation of a random sample of 100 interactions.
   • CAutomated evaluation using a custom metric that compares recommendations to actual purchases.
   • DA/B testing with live customers over a month.
   Reveal Solution Discussion Select an answer first
4. 28application · medium

   A healthcare organization is building an AI agent that answers patient questions about insurance coverage. They have a small set of 50 real patient queries with verified answers from their call center. They want to measure how accurately the agent handles these queries before deployment. Which approach should they use?

   • ACreate a test set from the 50 real queries and run a manual evaluation comparing agent responses to the verified answers.
   • BDeploy the agent to a small group of patients and ask them to rate the responses.
   • CGenerate 500 synthetic queries using an LLM and run an automated evaluation with LLM-as-a-judge.
   • DUse the 50 real queries as a test set and run an automated evaluation using exact-match scoring.
   Reveal Solution Discussion Select an answer first
5. 29application · medium

   A financial services company is evaluating an AI agent that answers customer questions about loan products. They have a test set of 200 questions with expected answers. They want to measure both the factual correctness of the answers and whether the agent's responses are safe and compliant with regulations. Which evaluation method should they use?

   • AUse an LLM-based evaluator to grade each response for correctness against the expected answer, and also run a separate safety/compliance evaluator that checks for prohibited content.
   • BMeasure only the agent's confidence scores and assume high confidence means the answer is correct and safe.
   • CManually review all 200 responses for correctness and safety, since automated evaluators are not reliable.
   • DRun the test set through the agent and use the built-in telemetry to see which questions were answered correctly.
   Reveal Solution Discussion Select an answer first
6. 30expert · hard

   A company has an AI agent that provides financial advice. They need to evaluate the agent for accuracy and regulatory compliance. They have a test set of 1,000 queries with verified answers. The compliance team has limited time. Which evaluation method is most appropriate?

   • AA/B testing with real customers.
   • BAutomated evaluation for accuracy, with compliance team review of only the responses that involve regulatory topics.
   • CManual evaluation of all 1,000 queries by the compliance team.
   • DAutomated evaluation using LLM-as-a-judge for both accuracy and compliance.
   Reveal Solution Discussion Select an answer first