What is involved in configuring a custom Scorer for integration into an evaluation workflow?

Deploying the Scorer to a production endpoint

Creating a Delta table to store Scorer results

Defining the scoring logic and specifying how it is invoked in the evaluation

Setting up a separate cluster for each Scorer

A team wants to evaluate their agent's responses for factual consistency with a provided context. They have a dataset of questions, contexts, and reference answers. They want to use a Databricks custom Scorer to compute a factual consistency score. What is the correct way to implement this?

Use the built-in answer similarity metric to measure factual consistency.

Use the model registry to compare the response against the reference answer.

Write a custom Scorer that takes the context and the agent's response, and returns a score based on factual alignment.

Use MLflow tracing to capture the context and response, then manually score each one.

An organization wants to evaluate their agent's responses for adherence to internal compliance guidelines. They have a set of rules that the responses must follow. They plan to use Databricks custom Scorers. What is the best way to implement this?

Create a custom Scorer that checks each response against the compliance rules and returns a pass/fail or score.

Use MLflow tracing to capture the responses and manually review them for compliance.

Use a built-in MLflow metric like answer similarity to infer compliance.

Store the compliance rules in the MLflow model registry and use them during inference.

A team is monitoring their agent in production. They notice that some responses are taking much longer than expected. They want to identify which part of the agent's execution is causing the delay. They have MLflow tracing enabled. What should they do?

Review the agent's prompt to see if it is too complex.

Use mlflow.evaluate() to compute a latency metric on the agent's outputs.

Inspect the trace spans to see the duration of each step, such as tool calls and LLM invocations.

Check the model registry for the latest version of the agent's model.

DatabricksCertified Generative AI Engineer Associate

Domain 6Objective 3

Agent Evaluation and Feedback GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 5)

Part of the Section 6: Evaluation and Monitoring domain, which makes up ~17% of our current practice bank. Databricks does not publish an official question count, but from its 90-minute exam (~35–60 total, ~6–10 in this domain), expect 1–2 from this objective — we provide 26 practice questions to prepare you well beyond it. (estimate)

26questions here

6free pages

6concepts

Questions 21–25

21foundation · easy
What is involved in configuring a custom Scorer for integration into an evaluation workflow?
Select an answer first
22application · medium
A team wants to evaluate their agent's responses for factual consistency with a provided context. They have a dataset of questions, contexts, and reference answers. They want to use a Databricks custom Scorer to compute a factual consistency score. What is the correct way to implement this?
Select an answer first
23application · medium
An organization wants to evaluate their agent's responses for adherence to internal compliance guidelines. They have a set of rules that the responses must follow. They plan to use Databricks custom Scorers. What is the best way to implement this?
Select an answer first
24application · medium
A team is monitoring their agent in production. They notice that some responses are taking much longer than expected. They want to identify which part of the agent's execution is causing the delay. They have MLflow tracing enabled. What should they do?
Select an answer first
25foundation · easy
Which MLflow component is specifically designed to apply scoring functions to agent outputs for performance analysis?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to GENERATIVE-AI-ENGINEER-ASSOCIATE

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by Databricks. “GENERATIVE-AI-ENGINEER-ASSOCIATE” is a trademark of its owner, used for identification only.