Agent Evaluation and Feedback GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   How are custom Scorers typically integrated into an evaluation workflow in Databricks?

   • ABy creating a separate notebook for each Scorer
   • BBy configuring them in the cluster's Spark configuration
   • CBy passing them as arguments to the evaluation function
   • DBy registering them in the Model Registry
   Reveal Solution Discussion Select an answer first
2. 7foundation · easy

   What is the primary purpose of using MLflow scoring functions in agent evaluation?

   • ATo store agent outputs in the MLflow tracking server
   • BTo trace the internal reasoning of the agent
   • CTo deploy the agent to a production endpoint
   • DTo compute evaluation metrics on agent outputs
   Reveal Solution Discussion Select an answer first
4. 8expert · medium

   A team is debugging an agent that sometimes fails to call a required tool. They have MLflow tracing enabled. They also have SME feedback indicating that the agent's responses are often incomplete. They want to correlate the missing tool calls with the incomplete responses. What is the best approach?

   • AUse mlflow.evaluate() to compute a metric that measures tool call frequency.
   • BReview the agent's prompt to see if the tool call instruction is clear.
   • CInspect the MLflow traces for the failing cases to see if the tool call was skipped, and correlate with the SME feedback on those specific responses.
   • DUse the model registry to compare different versions of the agent to see which one has the issue.
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A team is evaluating an agent that performs multi-step reasoning. They want to compute a metric that measures the correctness of the final answer. They have a golden dataset with correct answers. They also want to capture the reasoning traces for later analysis. What should they do?

   • AUse MLflow tracing only, and manually compute the correctness metric from the traces.
   • BUse mlflow.evaluate() to compute the correctness metric, and enable MLflow tracing to capture the reasoning traces.
   • CUse the model registry to compare the final answers with the golden dataset.
   • DUse mlflow.evaluate() only, and rely on the logged metrics to understand the reasoning process.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A healthcare company is developing an agent that answers patient questions. They want to ensure the responses are medically accurate. They have a team of doctors who can review the agent's responses. What is the best way to collect and use their feedback?

   • AHave the doctors review a sample of responses and provide detailed feedback on accuracy, which is then used to update the agent's prompt and evaluation criteria.
   • BUse automated metrics like answer similarity to evaluate accuracy, without involving the doctors.
   • CUse MLflow tracing to analyze the agent's reasoning and infer accuracy.
   • DCollect feedback from patients instead of doctors, as they are the end users.
   Reveal Solution Discussion Select an answer first