Monitoring, Testing, and Troubleshooting AI Solutions PROFESSIONAL-MACHINE-LEARNING-ENGINEER Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   Which of the following is a key aspect of monitoring generative AI solutions?

   • AMonitoring model training loss
   • BEnsuring outputs are safe and aligned with expected behavior
   • CTracking changes in feature distributions
   • DChecking for training-serving skew
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   Which testing strategy involves intentionally providing adversarial inputs to a generative AI model to uncover vulnerabilities?

   • ABenchmark testing
   • BPrompt testing
   • CAdversarial testing
   • DOutput consistency testing
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   A fintech company's fraud detection model was trained when a transaction over $500 was considered high-risk. Six months after deployment, the model's precision has dropped significantly. The input feature distributions (transaction amount, frequency, location) remain identical to the training data. What is the most likely cause of the performance degradation?

   • AFeature attribution drift caused by the model relying on a different set of features.
   • BTraining-serving skew caused by a change in the data preprocessing pipeline.
   • CData drift caused by a change in the underlying population of transactions.
   • DConcept drift caused by a change in the relationship between features and the fraud label.
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A credit risk model has been in production for a year. The monitoring dashboard shows that the input feature distributions (income, debt-to-income ratio) are stable, but the model's AUC has dropped from 0.85 to 0.72. The feature attribution scores have also changed significantly. What is the most likely explanation, and what should the ML engineer do?

   • AConcept drift is occurring; retrain the model with recent labeled data.
   • BData drift is occurring; retrain the model with recent data.
   • CFeature attribution drift is the root cause; adjust the model's decision threshold.
   • DTraining-serving skew is occurring; fix the data preprocessing pipeline.
   Reveal Solution Discussion Select an answer first
6. 20expert · hard

   A model predicting housing prices was trained on data from 2015-2020. In 2023, the model's performance has degraded. The monitoring dashboard shows that the distribution of 'square footage' has shifted (newer homes are larger), and the relationship between 'square footage' and price has also changed (larger homes now command a higher premium). What is happening, and what is the best response?

   • AOnly data drift is occurring; retrain the model with recent data.
   • BBoth data drift and concept drift are occurring; retrain the model with recent data.
   • COnly concept drift is occurring; retrain the model with recent data.
   • DTraining-serving skew is occurring; fix the data preprocessing pipeline.
   Reveal Solution Discussion Select an answer first