Evaluate Agent Performance AB-620 Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   After an evaluation, an AI agent builder sees that the agent performs well on short queries but poorly on long, complex queries. The test set contains a mix of both. The builder needs to improve the agent's performance on complex queries. What should they do first?

   • ARetrain the agent on a larger dataset of complex queries.
   • BAnalyze the errors on complex queries to identify common patterns, such as missing context or misunderstanding multi-step instructions.
   • CReduce the complexity of the agent's responses to make them shorter.
   • DAdd more complex queries to the test set to increase the accuracy score.
   Reveal Solution Discussion Select an answer first
2. 12expert · hard

   A company has an AI agent that provides customer support. They have a test set of 1,000 queries. The evaluation shows 98% accuracy, but a manual review of a random sample reveals that the agent often gives correct answers but in a rude tone. The company has a strict policy that all customer interactions must be polite. The team is under pressure to release the agent. What should they do?

   • ARelease the agent and plan to fix tone issues in a future update based on user feedback.
   • BReduce the test set to only queries where the agent is polite, so the accuracy score remains high.
   • CAdd tone evaluation to the automated test set and refine the agent's instructions to enforce a polite tone before release.
   • DRelease the agent because accuracy is high and tone is subjective.
   Reveal Solution Discussion Select an answer first
4. 13expert · hard

   A company has an AI agent that provides HR policy answers. They need to evaluate the agent's accuracy and compliance with company policies. They have a test set of 100 queries with verified answers. The team is small and has limited time. Which evaluation approach balances accuracy and effort?

   • ASkip evaluation and rely on the agent's training data.
   • BUse only manual evaluation for all 100 queries.
   • CUse only automated evaluation with LLM-as-a-judge.
   • DUse a hybrid approach: automated evaluation for factual accuracy and manual review for compliance-sensitive responses.
   Reveal Solution Discussion Select an answer first
5. 14expert · hard

   An AI agent for a government agency was evaluated, and the results show that the agent is 99% accurate on the test set. However, in production, users report that the agent often gives incorrect information. What is the most likely reason for this discrepancy?

   • AThe evaluation method was too lenient.
   • BThe agent's temperature is set too high in production.
   • CThe production environment has a different model version.
   • DThe test set does not represent the real-world distribution of user queries.
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A team is building a test set for an AI agent that handles product returns. They have collected 500 real user queries. They want to evaluate the agent's performance on different product categories and return reasons. They also want to ensure the test set is not too large to run frequently. What is the best approach?

   • ACreate a stratified sample of queries that ensures each product category and return reason is represented, and keep the test set to a manageable size.
   • BRandomly select 100 queries without considering category or reason.
   • CUse only queries from the most common product category to keep the test set small.
   • DInclude all 500 real queries in the test set to maximize coverage.
   Reveal Solution Discussion Select an answer first