Oversee AI/ML Model QA/QC (e.g., Configuration Management, Model Performance) PMI-CPMAI Practice Questions (Page 5)

Questions 21–25

1. 21application · easy

   A team is developing a model for a client. The client requires that the model be reproducible and that all changes be traceable. What is the most important practice to implement?

   • AUse a version control system for code and data, and log all experiments
   • BAssign a single developer to manage all model changes
   • CUse a cloud-based notebook for all development
   • DDocument the model architecture in a Word document
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A technical review is being conducted for a new recommendation model. The reviewer notices that the model uses a feature that is highly correlated with the target variable but is only available after the recommendation is made. What should the reviewer do?

   • ASuggest the team deploy the model and monitor the feature's availability
   • BDocument the feature as a limitation and proceed with deployment
   • CFlag the feature as a data leakage issue and require its removal or replacement
   • DApprove the model because the feature improves accuracy
   Reveal Solution Discussion Select an answer first
4. 23expert · hard

   A team is conducting a technical review of a model that will be used to predict equipment failure. The model uses a feature that is highly predictive but is only available after the failure occurs. The team is considering using this feature because it dramatically improves accuracy. What should the technical review conclude?

   • AApprove the model but document the feature as a limitation
   • BReject the model and require the team to remove the feature or find a proxy that is available at prediction time
   • CApprove the model because the feature improves accuracy and the business wants the best model
   • DSuggest the team deploy the model and monitor its performance
   Reveal Solution Discussion Select an answer first
5. 24application · medium

   A team is monitoring a model's performance during development. They notice that the F1 score on the validation set has been steadily increasing, but the F1 score on the test set has plateaued. What should the team do?

   • ASwitch to the test set for hyperparameter tuning to improve its score
   • BStop training and investigate the discrepancy between validation and test performance
   • CContinue training because the validation score is still improving
   • DIgnore the test set and rely on the validation set for model selection
   Reveal Solution Discussion Select an answer first
6. 25application · medium

   A data scientist has written a model training script that is a single, long file with no comments and many hard-coded values. The team's coding standards require modularity and documentation. What should the QA lead do?

   • AAsk the data scientist to add comments only at the top of the file
   • BRequire the data scientist to refactor the script into modular functions with docstrings and configuration files
   • CAllow the script to be used but require a peer review of the results
   • DApprove the script because it produces correct results
   Reveal Solution Discussion Select an answer first