Agent Evaluation and Feedback GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   How can SME feedback be used to improve an agent's performance?

   • ABy refining the agent's prompts based on feedback
   • BBy ignoring feedback that conflicts with automated metrics
   • CBy increasing the agent's response length
   • DBy automatically retraining the agent on all feedback
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   What is the main purpose of MLflow tracing in agent evaluation?

   • ATo capture and inspect the execution traces of an agent
   • BTo compute aggregate metrics like accuracy
   • CTo store training data for the agent
   • DTo deploy the agent to a serving endpoint
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A company is deploying an agent that answers employee HR questions. They want to improve the agent's responses based on feedback from HR specialists. What is the most effective way to collect and use this feedback?

   • AHave HR specialists review a sample of agent responses and provide feedback through a structured interface that records their ratings and comments.
   • BUse MLflow tracing to analyze the agent's execution traces and infer improvement areas without SME input.
   • CAutomatically use user thumbs-up/down ratings from the chat interface as the sole source of feedback.
   • DDeploy the agent and wait for complaints from employees to identify issues.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Which of the following is a common method for collecting SME feedback on agent responses?

   • AAutomatically deploying the agent to production
   • BUsing a random sampling of responses without review
   • CReviewing agent responses in a feedback interface
   • DRunning a load test on the agent
   Reveal Solution Discussion Select an answer first
6. 15application · medium

   A team is evaluating an agent that generates SQL queries from natural language. They have a golden dataset of natural language questions and correct SQL queries. They want to compute a metric that measures the syntactic correctness of the generated SQL. What should they do?

   • AUse MLflow's built-in answer similarity metric to compare the generated SQL to the correct SQL.
   • BUse the model registry to compare the generated SQL against the golden dataset.
   • CImplement a custom Scorer that parses the generated SQL and checks for syntax errors, then use it in mlflow.evaluate().
   • DUse MLflow tracing to capture the SQL generation process and manually inspect each query.
   Reveal Solution Discussion Select an answer first