Limitations of ML Functional Performance Metrics CT-AI Practice Questions (Page 5)

Questions 21–25

1. 21foundation · easy

   A model for predicting customer churn is evaluated on a test set from last year and achieves high accuracy. Why might this metric not reflect the model's current performance?

   • AAccuracy is not a valid metric for churn prediction because it is a regression problem.
   • BThe test set is too small to compute accuracy.
   • CCustomer behavior may have changed since last year, so the data distribution now may be different from the test set.
   • DThe model is overfitting because it was trained on last year's data.
   Reveal Solution Discussion Select an answer first
2. 22foundation · easy

   Why does ROC-AUC not reflect the performance at a specific operating point?

   • AROC-AUC is computed only at the default threshold of 0.5.
   • BROC-AUC summarizes performance across all possible thresholds, so it does not indicate the precision and recall at any single threshold.
   • CROC-AUC is a measure of calibration, not discrimination.
   • DROC-AUC is only valid for balanced datasets.
   Reveal Solution Discussion Select an answer first
4. 23expert · hard

   A team is building a model to detect rare diseases. They have two models: Model X has precision=0.90 and recall=0.50; Model Y has precision=0.50 and recall=0.90. The clinical team wants to ensure that no patient with the disease is missed, but they also have limited resources for follow-up tests. Which model should they choose?

   • ANeither model, because they both have low F1-scores
   • BEither model, because they have the same F1-score
   • CModel X, because it has higher precision, which reduces unnecessary follow-up tests
   • DModel Y, because it has higher recall, which is critical for not missing cases
   Reveal Solution Discussion Select an answer first
5. 24expert · hard

   A team is comparing two models for detecting fraudulent transactions. Model A has a ROC-AUC of 0.95, and Model B has a ROC-AUC of 0.90. However, when they deploy the models with a fixed threshold that achieves 90% recall, Model B has higher precision than Model A. Which conclusion is MOST appropriate?

   • AModel B is always better because it has higher precision at the operating point
   • BROC-AUC does not reflect performance at the specific operating point, so Model B may be better for the deployment threshold
   • CModel A is always better because it has a higher ROC-AUC
   • DROC-AUC is not a valid metric for imbalanced datasets
   Reveal Solution Discussion Select an answer first
6. 25foundation · easy

   In a fraud detection system, the cost of missing a fraudulent transaction is much higher than the cost of a false alarm. Which limitation of accuracy is most relevant in this scenario?

   • AAccuracy does not account for unequal costs of different error types.
   • BAccuracy cannot be computed when the dataset is imbalanced.
   • CAccuracy is only valid for balanced datasets.
   • DAccuracy is not a valid metric for fraud detection because it is a classification problem.
   Reveal Solution Discussion Select an answer first