Specialized Testing Considerations CT-AI Practice Questions (Page 4)

Questions 16–20

1. 16application · medium

   A healthcare organization is testing an AI system that assists radiologists in detecting anomalies in X-ray images. During test execution, the radiologists often accept the AI's findings without reviewing the images themselves. The test team wants to implement a strategy to detect and mitigate automation bias during testing. Which approach would be most effective?

   • ADocument the AI's training data and model architecture in the test report.
   • BInclude test cases where the AI is intentionally incorrect and verify that human reviewers catch the errors.
   • CUse a separate test environment to avoid any interference with the production system.
   • DIncrease the number of test cases to improve the statistical confidence in the AI's accuracy.
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   Which metric is specifically designed to detect concept drift by comparing distributions?

   • AMean Absolute Error (MAE)
   • BF1-score
   • CPopulation Stability Index (PSI)
   • DArea Under the ROC Curve (AUC)
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   Which condition increases the risk that automation bias will cause incorrect decisions in an AI-based system?

   • AUsers are under time pressure and have limited ability to verify AI outputs.
   • BThe AI system is used only in low-stakes decision-making.
   • CThe AI system's outputs are always correct in the test environment.
   • DThe AI system provides a confidence score with every output.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   An e-commerce company uses an AI system to recommend products to customers. Over time, customer preferences shift due to changing trends. The AI's recommendation accuracy declines even though the input data distribution (e.g., the types of products viewed) remains similar. What is the most likely cause of this performance decline?

   • AModel drift, because the model's parameters have changed over time.
   • BData drift, because the distribution of input features has changed over time.
   • COverfitting, because the model has memorized the training data too well.
   • DConcept drift, because the relationship between the input features and the target variable (what customers want) has changed.
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A bank deploys an AI system that approves or rejects loan applications. The system has been highly accurate in production for six months. During regression testing, a tester notices that the AI's decisions are rarely questioned by human reviewers, even when the AI rejects applications that clearly meet the bank's stated criteria. The reviewers say they trust the AI because it has been correct in the past. What is the most significant risk in this scenario?

   • AThe test data used for regression testing is not representative of the production data.
   • BThe human reviewers are exhibiting automation bias, which can lead to incorrect decisions being accepted without proper scrutiny.
   • CThe AI system is suffering from concept drift because its performance has changed over time.
   • DThe AI system is overfitting to historical loan applications, causing it to reject valid applications.
   Reveal Solution Discussion Select an answer first