Challenges Testing Self-Learning Systems CT-AI Practice Questions (Page 4)

Questions 16–20

1. 16application · medium

   A team is testing a self-learning system that recommends products based on user behavior. The system was performing well in testing, but after deployment, its accuracy on the live data is significantly lower than expected. The team suspects data drift. What is the most effective way to validate this hypothesis?

   • ARetrain the model on the live data and re-run the test suite
   • BCompare the statistical distribution of the live input data with the training data used during testing
   • CRe-run the original test suite against the live system and compare the results
   • DCheck the system's logs for any errors or exceptions during the deployment process
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A team has deployed a self-learning system that recommends news articles. They are monitoring its performance using click-through rate (CTR) as the primary metric. After a few weeks, the CTR has increased, but user engagement time has decreased. The team is concerned that the system is optimizing for clicks on sensationalist content. What is the most appropriate action?

   • AIncrease the frequency of the system's retraining to adapt faster
   • BModify the system's reward function to include engagement time as a secondary objective
   • CAccept the increased CTR as a sign of success and continue monitoring
   • DRevert to the previous version of the system
   Reveal Solution Discussion Select an answer first
4. 18expert · hard

   A team is testing a self-learning system for stock trading. The system uses reinforcement learning and is non-deterministic. The team needs to validate that the system is profitable. They run the system in a simulation with the same initial conditions multiple times and get different profit results. What is the most appropriate way to evaluate the system's performance?

   • ATake the average profit across all runs and use that as the performance metric
   • BAnalyze the distribution of results, including the mean, variance, and worst-case scenario
   • CUse the best result as the performance metric, as it shows the system's potential
   • DRun the system once with a fixed random seed to get a deterministic result
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A team is testing a self-learning system that moderates content on a social media platform. The system learns from user reports and moderator actions. The team is concerned about the system's potential to censor legitimate speech or allow harmful content. What is the most important testing consideration?

   • AEnsuring the system can process a high volume of content quickly
   • BEnsuring the system's decisions are consistent over time
   • CVerifying that the system's decisions are explainable to users
   • DBalancing the system's sensitivity to detect harmful content with its specificity to avoid over-censorship
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A team is testing a self-learning system that predicts equipment failure in a factory. The system is trained on sensor data from the machines. After a major upgrade to the machines, the system's prediction accuracy drops significantly. What is the most likely cause?

   • AThe system's code has a bug that was introduced during the upgrade
   • BThe test environment is not configured correctly for the new machines
   • CThe system has forgotten how to predict failures due to a lack of new training data
   • DThe sensor data distribution has changed due to the new machines, causing data drift
   Reveal Solution Discussion Select an answer first