Limitations of ML Functional Performance Metrics CT-AI Practice Questions (Page 6)

Questions 26–30

1. 26application · medium

   A medical imaging team develops a model to detect a rare disease that occurs in only 2% of the population. The model is tuned to have high precision (0.95) to minimize false alarms, but its recall is only 0.40. The clinical team wants to ensure that most actual cases are not missed. Which metric would BEST highlight the trade-off the team is facing?

   • ARecall alone
   • BPrecision alone
   • CBoth precision and recall reported together
   • DF1-score
   Reveal Solution Discussion Select an answer first
2. 27application · medium

   A model predicts whether a customer will buy a product. The overall F1-score is 0.85. However, the model performs much better on customers from urban areas (F1=0.90) than on rural customers (F1=0.60). Which limitation of aggregate metrics does this illustrate?

   • ASingle-number metrics can obscure performance differences across data segments
   • BF1-score is threshold-dependent
   • CF1-score assumes equal importance of precision and recall
   • DF1-score is not suitable for imbalanced datasets
   Reveal Solution Discussion Select an answer first
4. 28application · medium

   A model predicts customer churn. The overall confusion matrix shows 90% accuracy. However, the model performs poorly on a specific subgroup: customers aged 18-25, where it misclassifies 70% of churners. Which limitation of confusion matrix metrics does this scenario illustrate?

   • AThey are sensitive to class distribution and may not be comparable across datasets
   • BThey do not reflect performance when data distribution shifts over time
   • CThey are threshold-dependent and can vary with the decision threshold
   • DThey provide aggregate views that can hide per-subgroup performance issues
   Reveal Solution Discussion Select an answer first
5. 29expert · hard

   A model predicts loan default risk. It is trained on historical data from 2015-2020 and achieves an AUC of 0.85 on a test set from 2020. In 2021, the economy enters a recession, and the default rate increases significantly. The model's performance on new data drops. Which action would BEST address the limitation of metrics in non-stationary environments?

   • ASwitch to a threshold-independent metric like ROC-AUC
   • BIgnore the new data because the model was validated on historical data
   • CContinuously monitor metrics on new data and retrain the model periodically
   • DUse a larger test set from 2020 to improve the AUC estimate
   Reveal Solution Discussion Select an answer first
6. 30foundation · easy

   In a highly imbalanced dataset where the positive class is rare, which limitation of precision is most relevant?

   • APrecision is not computable when the dataset is imbalanced.
   • BPrecision can be high even if the model misses many actual positives.
   • CPrecision is always higher than recall in imbalanced datasets.
   • DPrecision does not consider the number of false negatives.
   Reveal Solution Discussion Select an answer first