Limitations of ML Functional Performance Metrics CT-AI Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   Which statement best describes a limitation of ROC-AUC?

   • AROC-AUC is not affected by the choice of decision threshold.
   • BROC-AUC is only defined for binary classification.
   • CROC-AUC can be overly optimistic when the dataset is imbalanced because the false positive rate is dominated by the large number of true negatives.
   • DROC-AUC is always lower than accuracy.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A model predicts house prices. The evaluation report shows MAE=50,000 and RMSE=200,000. The data scientist notes that a few luxury homes have prices around $5 million, while most homes are under $500,000. Which limitation of regression metrics does this scenario illustrate?

   • ARMSE is scale-dependent and cannot be compared across datasets
   • BRMSE is more sensitive to outliers than MAE, so the large difference indicates the presence of outliers
   • CMAE does not capture any error information
   • DMAE is always better than RMSE for skewed data
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A model predicts customer churn for a telecom company. It achieves 85% accuracy on a test set collected last year. After deployment, the company launches a new pricing plan that changes customer behavior. The model's real-world performance drops significantly. Which limitation of metrics does this illustrate?

   • AAccuracy is misleading in imbalanced datasets
   • BF1-score assumes equal importance of precision and recall
   • CConfusion matrix metrics hide per-class performance issues
   • DMetrics computed on static test sets may not reflect performance when data distribution shifts over time
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   A binary classifier for a rare disease achieves 95% accuracy on a test set where only 5% of cases are positive. Which statement best explains why accuracy alone is misleading here?

   • AA model that predicts the majority class for every instance would also achieve 95% accuracy, so accuracy does not reflect the model's ability to detect the positive class.
   • BAccuracy cannot be computed when the test set is imbalanced.
   • CAccuracy is not a valid metric because it does not consider the cost of false positives.
   • DAccuracy is always unreliable for any medical diagnosis model.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   Which statement best describes a limitation of threshold-dependent metrics like accuracy, precision, and recall?

   • AThese metrics are always higher when the threshold is set to 0.5.
   • BThese metrics can change significantly with the chosen decision threshold, so the same model can appear to have very different performance depending on the threshold.
   • CThese metrics are not affected by the class distribution.
   • DThese metrics are only defined for binary classification.
   Reveal Solution Discussion Select an answer first