Specialized Testing Considerations CT-AI Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A team is maintaining an AI system that predicts equipment failure in a manufacturing plant. A new engineer joins the team and needs to understand how the AI model was trained and why it makes certain predictions. Which documentation practice would most help the new engineer?

   • AMaintaining a detailed model card that includes training data, evaluation results, and intended use cases.
   • BKeeping a log of all changes made to the model's code in a version control system.
   • CStoring the raw training data in a shared drive for the engineer to analyze.
   • DProviding the engineer with access to the model's source code and letting them figure it out.
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A company has an AI system that has been in production for several years. The team needs to update the system to comply with new regulations. Which documentation practice would most support the traceability of changes to the AI component?

   • AKeep a single document that is updated whenever changes are made.
   • BStore all training data in a shared folder without any versioning.
   • CMaintain a versioned model registry that records each model version, its training data, and its performance metrics.
   • DUse email to communicate changes to the team.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A team is testing an AI system that assists with legal document review. The testers are required to manually review a sample of the AI's decisions to ensure accuracy. However, they often skip this review because the AI has been reliable. Which testing strategy would best mitigate the risk of automation bias in this scenario?

   • AImplement a mandatory second review for a random sample of cases, with the reviewer blinded to the AI's decision.
   • BDocument the AI's performance metrics in the test report.
   • CProvide the testers with more training on how the AI works.
   • DIncrease the sample size of the manual review to cover more cases.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Why is automation bias particularly relevant when testing AI-based systems?

   • ABecause AI systems are more prone to automation bias than human testers.
   • BBecause AI systems are always correct and testers should trust their outputs.
   • CBecause testers may unconsciously accept AI outputs without sufficient verification, leading to undetected defects.
   • DBecause automation bias causes AI models to produce biased predictions.
   Reveal Solution Discussion Select an answer first
6. 15application · medium

   An AI system that predicts customer churn has been in production for two years. The test team suspects concept drift is occurring because the model's accuracy has declined. To evaluate the impact of concept drift on the system's reliability, which testing approach is most appropriate?

   • ARetrain the model on the most recent data and deploy it immediately.
   • BIncrease the frequency of model monitoring to detect drift earlier.
   • CReview the model's training data to see if it is still representative.
   • DRun the model on recent production data and compare its performance to its performance on historical test data.
   Reveal Solution Discussion Select an answer first