Monitoring, Testing, and Troubleshooting AI Solutions PROFESSIONAL-MACHINE-LEARNING-ENGINEER Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   Which monitoring approach directly addresses training-serving skew?

   • ATracking model prediction latency
   • BReviewing model training logs
   • CMonitoring model accuracy on a held-out test set
   • DComparing feature distributions between training and serving data
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A team is developing a Gemini-powered code-generation assistant. Before release, they want to test whether the model produces consistent outputs when given the same prompt multiple times and whether it handles prompts designed to trick it into generating insecure code. Which testing approach should they use?

   • ARun the same prompt multiple times and measure output consistency, then run adversarial prompts targeting security.
   • BDeploy the model to a staging environment and monitor user interactions for errors.
   • CMeasure the model's inference latency on a standard set of prompts.
   • DEvaluate the model's perplexity on a large corpus of code snippets.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   An enterprise has deployed a Gemini-based agent on the Gemini Enterprise Agent Platform. The ML team wants to establish continuous evaluation metrics for the production model, including detecting data drift and concept drift. What should they configure?

   • ASet up Model Monitoring on the Gemini Enterprise Agent Platform with drift detection enabled.
   • BEnable Vertex AI TensorBoard to visualize training metrics in real time.
   • CCreate a batch prediction job that runs weekly and compares outputs to a golden dataset.
   • DDeploy a separate model to shadow the production model and compare their predictions.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   A credit risk model was trained when interest rates were low. Now, with higher interest rates, the same feature values lead to different default probabilities. This is an example of:

   • AConcept drift
   • BData drift
   • CFeature attribution drift
   • DTraining-serving skew
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A model predicting customer lifetime value has been stable in terms of accuracy and input distributions for six months. However, the Model Monitoring dashboard on the Gemini Enterprise Agent Platform shows that feature attribution scores have shifted dramatically: 'purchase frequency' was the top feature, but now 'customer age' is the top feature. The business team is concerned because the model's decisions seem less explainable. What is the best course of action?

   • ASwitch to a simpler model that is inherently more explainable.
   • BRetrain the model immediately to restore the original feature attributions.
   • CInvestigate the cause of the feature attribution drift, including checking for hidden changes in the data or model behavior.
   • DIgnore the change because accuracy and input distributions are stable.
   Reveal Solution Discussion Select an answer first