Monitor and Ensure Operational Performance CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A data science team has deployed a churn prediction model that consumes customer data from a data warehouse. The model's performance is evaluated weekly, but the team wants to detect issues earlier. They plan to set up continuous monitoring of the pipeline. Which combination of checks should they implement to ensure both data quality and model performance are tracked?

   • AMonitor only the data pipeline's execution time and failure rates.
   • BMonitor data freshness, schema changes, and feature distribution drift, along with model accuracy on a rolling window.
   • CMonitor the model's training data distribution and retrain automatically when drift is detected.
   • DMonitor only the model's accuracy on a daily basis.
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   Which of the following is a business KPI that could be used to monitor a recommendation model's performance over time?

   • AData storage size
   • BNumber of model parameters
   • CClick-through rate on recommended items
   • DModel training time
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A credit card fraud detection model was trained on transaction data from 2023. In production, the model's precision has dropped from 0.85 to 0.60 over the past month, while recall has remained stable. The data science team suspects the distribution of transaction amounts has shifted. Which monitoring approach would best confirm this hypothesis?

   • ATrack the KL divergence between the training and live distributions of transaction amount features.
   • BMonitor the model's AUC-ROC score on a rolling window of live predictions.
   • CReview the model's feature importance scores to see if the ranking has changed.
   • DCompare the model's predicted probabilities against the actual fraud labels using a calibration plot.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Which of the following is a key component of a pipeline monitoring setup?

   • ALogging model predictions and input data
   • BChanging the model's algorithm
   • CIncreasing the model's learning rate
   • DReducing the number of features
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   Which technique is commonly used to detect data drift in a production model?

   • AChecking the model's code for bugs
   • BMonitoring the model's training loss
   • CReviewing the model's confusion matrix on the test set
   • DComparing the distribution of live input features to the training data distribution
   Reveal Solution Discussion Select an answer first