Test Oracles for AI-Based Systems CT-AI Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   An e-commerce company is testing an AI system that predicts the probability of a customer clicking on an ad. The system outputs a probability between 0 and 1. The team has historical click-through rates for different customer segments. They want to validate that the system's predictions are well-calibrated. Which oracle approach is most suitable?

   • AA statistical oracle that compares the average predicted probability to the actual click rate within each segment.
   • BA heuristic oracle that flags any probability above 0.9 as suspicious.
   • CA model-based oracle that compares the predictions to a simple logistic regression model trained on the same data.
   • DA human oracle where marketing experts judge whether each probability seems reasonable.
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   What is a primary limitation of using human experts as test oracles for AI systems?

   • AHigh consistency across different experts
   • BInability to judge subjective outputs
   • CLow cost for large-scale testing
   • DPoor scalability to large numbers of test cases
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   What is a common strategy for combining multiple oracles in AI testing?

   • AUsing only heuristic oracles because they are always correct
   • BUsing a model-based oracle to replace all other oracle types
   • CUsing a statistical oracle to validate all outputs and a human oracle to validate a subset
   • DUsing the most expensive oracle for all test cases
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Why might a tester combine a heuristic oracle with a human-based oracle?

   • ATo ensure that the oracle is always correct
   • BTo use the heuristic oracle for initial screening and the human oracle for ambiguous cases
   • CTo reduce the cost of testing by eliminating the need for human experts
   • DTo make the test oracle more complex and difficult to understand
   Reveal Solution Discussion Select an answer first
6. 15application · medium

   A robotics company is testing an AI system that controls a robot arm's movement. They have a high-fidelity physics simulator that accurately models the robot's behavior. They want to test the AI's control commands in a simulated environment before real-world deployment. Which oracle approach is most appropriate?

   • AA model-based oracle that uses the physics simulator to compute the expected outcome of the control commands and compares it to the AI's predicted outcome.
   • BA human oracle where an engineer watches the simulated movement and judges if it looks correct.
   • CA statistical oracle that checks if the distribution of joint angles matches historical successful movements.
   • DA heuristic oracle that checks if the robot arm stays within predefined joint limits.
   Reveal Solution Discussion Select an answer first