Evaluate Agent Performance AB-620 Practice Questions (Page 7)

Questions 31–35

1. 31expert · hard

   A company is creating a test set for an AI agent that handles multilingual customer support. They have real queries in English and Spanish, but not in French. They want to evaluate the agent's French performance. What should they do?

   • ADeploy the agent to French users and monitor performance in production.
   • BSkip French evaluation because the agent was trained on multilingual data.
   • CUse only English and Spanish queries and assume French performance is similar.
   • DGenerate synthetic French queries and have native speakers review them for accuracy.
   Reveal Solution Discussion Select an answer first
2. 32expert · hard

   An AI agent for a university was evaluated with a test set of 500 queries. The results show that the agent is 95% accurate on average, but it fails on all queries about financial aid. The test set contains only 20 financial aid queries. What is the most appropriate action?

   • ARemove financial aid queries from the test set to improve the average score.
   • BRetrain the agent on the entire test set.
   • CExpand the test set with more financial aid queries and re-evaluate to understand the failure pattern.
   • DAccept the 95% accuracy rate as sufficient.
   Reveal Solution Discussion Select an answer first
4. 33application · medium

   An AI agent builder runs an evaluation and sees that the agent's responses are factually correct but often include extra information that confuses users. The evaluation report shows high accuracy but low user satisfaction. What should the builder do next?

   • AReview the evaluation results for response quality metrics such as conciseness or relevance, and adjust the agent's instructions to be more concise.
   • BAdd more test cases that require longer responses to improve the accuracy score.
   • CIgnore user satisfaction and focus only on factual accuracy, since that is the primary goal.
   • DReplace the agent with a simpler rule-based system that gives shorter answers.
   Reveal Solution Discussion Select an answer first
5. 34application · medium

   A retail company has an AI agent that handles order returns. They have a test set of 500 customer queries with expected answers. They need to measure how often the agent provides the correct final answer and also how well the agent's responses align with the company's tone guidelines. Which evaluation approach should they use?

   • AUse the agent's confidence scores to determine which responses are correct and assume tone is acceptable if confidence is high.
   • BRun an automated evaluation that compares the agent's answers to the expected answers for accuracy, and use a separate LLM-based evaluator to score tone alignment.
   • CAsk the agent to self-evaluate its own responses for accuracy and tone during the test run.
   • DRun only the automated accuracy evaluation and manually spot-check tone on a few responses.
   Reveal Solution Discussion Select an answer first
6. 35application · medium

   A company has an AI agent that provides IT support. They have a test set of 300 queries with expected answers. They want to evaluate the agent's performance in a way that closely mimics how users will interact with it, including follow-up questions. Which evaluation method should they choose?

   • AUse a multi-turn evaluation that simulates a conversation flow, including follow-up questions and context.
   • BRun a single-turn evaluation where each query is tested independently.
   • CEvaluate only the first response of each conversation, since follow-ups are less important.
   • DUse a random sample of queries without expected answers and have the agent self-score its responses.
   Reveal Solution Discussion Select an answer first