Oversee AI Solution Metrics (e.g., KPI, Model Performance) PMI-CPMAI Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   An online retailer has a product-recommendation model. The data science team has set an alert that triggers when the model's click-through rate (CTR) drops by more than 15% from the 7-day rolling average. The team receives an alert, but upon investigation, they find that the drop was caused by a temporary website outage, not a model issue. What should the team do to prevent this type of false alert in the future?

   • AIgnore the alert and wait for the next one, since this was a one-time event.
   • BRetrain the model, since the CTR drop indicates the model is not working.
   • CAdjust the alert to exclude periods when the website is down, by integrating the alerting system with the website's uptime monitoring.
   • DIncrease the alert threshold to 25% to make it less sensitive.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A media company has a content-recommendation model. The product manager wants to be alerted when the model's engagement rate (click-through rate) drops below a certain level, but only if the drop is sustained for at least 24 hours, to avoid alert fatigue from temporary fluctuations. What should the alerting system be configured to do?

   • ATrigger an alert when the engagement rate drops below the threshold on average over a 24-hour period, even if it recovers in between.
   • BTrigger an alert immediately when the engagement rate drops below the threshold, regardless of duration.
   • CTrigger an alert only when the engagement rate has been below the threshold for a continuous 24-hour period.
   • DTrigger an alert when the engagement rate drops below the threshold for any 6-hour period, and escalate if it continues.
   Reveal Solution Discussion Select an answer first
4. 3expert · medium

   A telecom company has a churn-prediction model. The marketing team wants to see the number of customers predicted to churn each week, while the data science team wants to see the model's precision and recall trends. The two teams have conflicting needs: marketing wants a simple business view, and data science wants a detailed technical view. What is the best way to satisfy both?

   • ACreate a single dashboard with both churn count and precision/recall trends on the same page.
   • BCreate two separate dashboards: one for marketing with the churn count and one for data science with precision and recall trends.
   • CCreate a single dashboard showing only the churn count, and email the precision/recall trends to data science weekly.
   • DCreate a single dashboard showing only precision and recall, and have marketing use the raw data to calculate churn count.
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A manufacturing company has a quality-control model that detects defective products. The plant manager wants a weekly report that shows the number of defective products caught by the model and the cost savings from avoiding shipping defects. The data science team wants to include the model's precision and recall. What is the best way to structure the report?

   • AA report that only includes the business KPIs (defects caught, cost savings), since the plant manager does not understand technical metrics.
   • BA single table with all metrics mixed together, so the reader can see everything at once.
   • CA one-page executive summary with the business KPIs (defects caught, cost savings) at the top, followed by a technical section with precision and recall.
   • DA report that only includes the technical metrics (precision and recall), since they are the most important.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   What is the primary purpose of analyzing model performance trends over time?

   • ATo replace the need for real-time monitoring
   • BTo increase the frequency of model retraining
   • CTo compare the model against competitor models
   • DTo identify gradual degradation or drift in model performance
   Reveal Solution Discussion Select an answer first