What does the area under the ROC curve (AUC) measure for a binary classifier?

The probability that the model ranks a randomly chosen positive instance higher than a randomly chosen negative instance

The optimal threshold that maximizes the true positive rate

The overall accuracy of the model at a fixed threshold

The ratio of true positives to false positives at the default threshold

A data scientist compares two models using 10-fold cross-validation. The mean accuracy of Model A is 0.82, and Model B is 0.80. The paired t-test yields a p-value of 0.06. What is the most appropriate conclusion?

Model A is significantly better because its mean accuracy is higher

There is not enough evidence to conclude Model A is better than Model B at the 0.05 significance level

The models are identical because the p-value is not significant

Model B is better because the p-value is close to significance

Why is it important to evaluate a model on a held-out test set that was not used during training or validation?

To ensure the model is overfitted to the training data

To increase the model's accuracy

To get an unbiased estimate of how the model will perform on new, unseen data

Certified Artificial Intelligence Practitioner (CAIP)

Domain 3Objective 4

Objective 3.4 Evaluate the Model AIP-210 Practice Questions (Page 2)

Part of the 3.0 Training and Tuning ML Systems and Models domain, which accounts for 24% of the AIP-210 exam.

37questions here

8free pages

10concepts

24%of the exam

Questions 6–10

6foundation · easy
What does the area under the ROC curve (AUC) measure for a binary classifier?
Select an answer first
7expert · hard
A data scientist compares two models using 10-fold cross-validation. The mean accuracy of Model A is 0.82, and Model B is 0.80. The paired t-test yields a p-value of 0.06. What is the most appropriate conclusion?
Select an answer first
8foundation · easy
Which metric is calculated as the number of true positives divided by the sum of true positives and false positives?
Select an answer first
9application · medium
A fraud detection model flags transactions as fraudulent. In a test set of 10,000 transactions, 100 are truly fraudulent. The model flags 150 transactions, of which 90 are truly fraudulent. The business requires that at least 85% of actual fraud cases be caught, but also wants to minimize false alarms. Which metric should the data scientist prioritize to evaluate the model against the business requirement?
Select an answer first
10foundation · easy
Why is it important to evaluate a model on a held-out test set that was not used during training or validation?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to AIP-210

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by CertNexus. “AIP-210” is a trademark of its owner, used for identification only.

Objective 3.4 Evaluate the Model AIP-210 Practice Questions (Page 2)

Questions 6–10

What does the area under the ROC curve (AUC) measure for a binary classifier?

A data scientist compares two models using 10-fold cross-validation. The mean accuracy of Model A is 0.82, and Model B is 0.80. The paired t-test yields a p-value of 0.06. What is the most appropriate conclusion?

Which metric is calculated as the number of true positives divided by the sum of true positives and false positives?

Why is it important to evaluate a model on a held-out test set that was not used during training or validation?

Keep studying — 3.0 Training and Tuning ML Systems and Models