Task 2.3: Analyze Model Performance MLA-C01 Practice Questions (Page 5)

Questions 21–25

1. 21foundation · easy

   Which metric is most appropriate to optimize when the cost of a false negative is very high, such as in cancer screening where missing a positive case is dangerous?

   • ASpecificity
   • BPrecision
   • CAccuracy
   • DRecall
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A machine learning engineer is training a deep neural network for image classification. After 50 epochs, the training accuracy is 99.5% and the validation accuracy is 88.2%. The validation accuracy has been fluctuating between 87% and 89% for the last 20 epochs, while training accuracy continues to improve. Which action is most appropriate?

   • AAdd more training data or apply regularization techniques such as dropout or weight decay.
   • BReduce the model's capacity by removing layers, which will increase validation accuracy.
   • CIncrease the learning rate to help the model escape the local minimum.
   • DTrain for more epochs to allow the model to converge further.
   Reveal Solution Discussion Select an answer first
4. 23expert · hard

   A research team is developing a model to predict equipment failure in a manufacturing plant. They are experimenting with different feature sets, hyperparameters, and data preprocessing steps. The team needs to ensure that their experiments are reproducible so that they can compare results across different runs and share findings with stakeholders. They are using SageMaker for training. Which approach is most appropriate?

   • AUse a fixed random seed and document the code version in the training script.
   • BUse SageMaker Experiments to track parameters, metrics, and artifacts for each training run.
   • CStore all training data and code in an S3 bucket with versioning enabled.
   • DManually record the parameters and metrics in a shared spreadsheet for each run.
   Reveal Solution Discussion Select an answer first
5. 24foundation · easy

   In Amazon SageMaker, what is a shadow variant?

   • AA variant that is always inactive
   • BA variant that is used for A/B testing and serves traffic to a subset of users
   • CA variant that receives a copy of production traffic but does not serve responses to users
   • DA variant that serves traffic only during off-peak hours
   Reveal Solution Discussion Select an answer first
6. 25expert · hard

   A company has a production model that predicts customer lifetime value. They have trained a new model that they believe is more accurate. They want to test the new model in production without impacting the user experience. They set up a shadow variant in SageMaker that receives a copy of the production traffic. After running the shadow variant for two weeks, they compare the performance of the two models. The production model has an RMSE of 120, and the shadow variant has an RMSE of 95. However, the shadow variant's predictions are, on average, 10% higher than the production model's predictions. What is the most appropriate conclusion?

   • AThe shadow variant should be rejected because RMSE is not a suitable metric for this problem.
   • BThe shadow variant is better and should be promoted to production immediately.
   • CThe production model is better because the shadow variant's predictions are biased.
   • DThe shadow variant has a lower RMSE, but the systematic difference in predictions should be investigated before promotion.
   Reveal Solution Discussion Select an answer first