Evaluate Agent Performance AB-620 Practice Questions (Page 8)

Questions 36–38

1. 36application · medium

   A financial services firm is creating a test set for an AI agent that handles loan applications. They want to ensure the test set covers different income levels, loan amounts, and credit scores. What is the most important consideration when building this test set?

   • AIncluding only edge cases to stress-test the agent.
   • BUsing only real customer data to ensure authenticity.
   • CIncluding a balanced representation of all relevant demographic and financial categories.
   • DMaking the test set as large as possible regardless of relevance.
   Reveal Solution Discussion Select an answer first
2. 37application · medium

   An AI agent for a travel agency was evaluated using a test set of 500 queries. The results show that the agent performs well on queries about flights but poorly on queries about hotel bookings. What is the most likely explanation?

   • AThe agent's temperature setting is too high.
   • BThe test set may have more flight queries than hotel queries, or the agent was trained on more flight data.
   • CThe test set is too small.
   • DThe evaluation method was flawed.
   Reveal Solution Discussion Select an answer first
3. 38application · medium

   A team is creating a test set for an AI agent that helps users reset their passwords. They have collected 100 real user queries. They want the test set to be representative of the different ways users ask for help. What is the best approach to build the test set?

   • AGenerate all test queries synthetically to ensure full control over the test set.
   • BSelect a subset of queries that covers each intent and includes multiple paraphrases, and add synthetic variations for edge cases.
   • CInclude all 100 real queries as-is, since they represent actual user language.
   • DUse only the most common queries to keep the test set small and fast to run.
   Reveal Solution Discussion Select an answer first