Agent Evaluation and Feedback GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   What is the purpose of a Databricks custom Scorer in agent evaluation?

   • ATo evaluate agent and LLM outputs using custom-defined metrics
   • BTo deploy agents to production
   • CTo trace agent execution steps
   • DTo store agent outputs in a database
   Reveal Solution Discussion Select an answer first
2. 17expert · medium

   A team is building an agent for a niche domain. They have limited SME availability. They want to collect feedback efficiently to improve the agent. What is the best strategy?

   • AHave SMEs review a strategically selected sample of responses that cover a variety of cases, including edge cases and failures.
   • BHave SMEs review only the responses that the automated metrics flag as low quality.
   • CHave SMEs review every single response to ensure complete coverage.
   • DRely solely on automated metrics and skip SME feedback.
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   After collecting feedback from subject matter experts, a team finds that the agent frequently provides overly verbose answers. They want to use this feedback to improve the agent. What is the most direct way to incorporate this feedback?

   • AIncrease the temperature parameter to make the model more deterministic.
   • BChange the underlying LLM to a larger model that might be more concise.
   • CIgnore the feedback because the agent is already performing well on automated metrics.
   • DUpdate the agent's prompt to instruct it to be more concise, then re-evaluate with the same feedback dataset.
   Reveal Solution Discussion Select an answer first
5. 19expert · medium

   A team is evaluating an agent that uses multiple tools to answer questions. They want to understand how the agent's tool usage affects the quality of the final answer. They have MLflow tracing enabled and a golden dataset with correct answers. What is the best way to analyze this?

   • AUse MLflow tracing only, and manually compare the traces with the correct answers.
   • BUse MLflow tracing to capture the tool usage for each query, then correlate the tool usage patterns with the correctness metric computed via mlflow.evaluate().
   • CUse the model registry to compare different versions of the agent to see which one uses tools better.
   • DUse mlflow.evaluate() to compute a metric that measures tool usage frequency.
   Reveal Solution Discussion Select an answer first
6. 20expert · medium

   A team is evaluating an agent that summarizes legal documents. They have a golden dataset with reference summaries. They want to compute both a semantic similarity score and a domain-specific metric that checks for the inclusion of key legal terms. They have already implemented a custom Scorer for the legal terms. What is the most efficient way to compute both metrics in a single evaluation run?

   • AUse the model registry to compare the summaries and infer the metrics.
   • BUse MLflow tracing to compute both metrics manually from the traces.
   • CRun two separate mlflow.evaluate() calls: one for the built-in metric and one for the custom Scorer.
   • DUse mlflow.evaluate() with the built-in answer similarity metric and pass the custom Scorer via the 'scorers' parameter.
   Reveal Solution Discussion Select an answer first