Evaluate Agent Performance AB-620 Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   Which of the following is a key characteristic of a well-constructed test set for an AI agent?

   • AIt includes only the most complex scenarios the agent might encounter
   • BIt is generated automatically from the agent's production logs without review
   • CIt is kept secret from the development team to avoid bias
   • DIt contains a diverse set of representative inputs and clearly defined expected outputs
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A team is building a test set for an AI agent that books travel. They want to ensure the test set covers different travel scenarios, such as flights, hotels, and car rentals, as well as different user intents like booking, canceling, and changing reservations. What is the most important consideration when creating this test set?

   • AUse only real user queries from the production logs, regardless of scenario distribution.
   • BFocus on the most complex scenarios, since they are the hardest for the agent.
   • CInclude only queries that the agent has previously answered correctly to ensure a high accuracy score.
   • DInclude an equal number of queries for each scenario and intent to ensure balanced coverage.
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   A legal firm is building an AI agent to answer client questions about case status. They have a small number of real case queries and want to create a test set that also covers hypothetical scenarios. What should they do?

   • ACombine real queries with carefully crafted synthetic queries that reflect potential client questions.
   • BUse only synthetic queries to ensure privacy.
   • CAsk lawyers to write all test queries from scratch.
   • DUse only real queries to avoid synthetic data bias.
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   An AI agent builder is reviewing evaluation results. The agent's accuracy is 85%, but the confidence scores for incorrect answers are just as high as for correct answers. The builder needs to decide whether to deploy the agent. What should they do?

   • ADeploy the agent and rely on user feedback to catch incorrect answers.
   • BLower the accuracy threshold to 85% so the agent meets the requirement.
   • CDeploy the agent because accuracy is above the 80% threshold set by the team.
   • DInvestigate the relationship between confidence and correctness, and consider calibrating the agent or adding a fallback for low-confidence responses.
   Reveal Solution Discussion Select an answer first
6. 20expert · hard

   A company is evaluating an AI agent that handles product returns. They have two test sets: one with 500 real customer queries and another with 500 synthetic queries. The agent scores 90% on the real set and 70% on the synthetic set. What is the most likely explanation?

   • AThe synthetic test set may include edge cases or phrasings not present in the real data, revealing weaknesses.
   • BThe real test set is too easy and should be replaced with the synthetic set.
   • CThe agent was overfitted to the real test set.
   • DThe evaluation method is inconsistent between the two sets.
   Reveal Solution Discussion Select an answer first