Test Oracles for AI-Based Systems CT-AI Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A company is developing an AI system that generates personalized fitness plans. They have a team of certified fitness trainers available to review outputs. The system will be tested on a large number of diverse user profiles. The team is concerned about consistency across trainers and the time required for review. Which approach best addresses these concerns?

   • AHave a single trainer review all outputs to ensure consistency.
   • BUse a panel of trainers to review a sample of outputs, and use a heuristic to check for mandatory components in the rest.
   • CUse a statistical oracle to compare the distribution of plan lengths and intensities across user profiles.
   • DUse a heuristic oracle that checks if the plan includes all mandatory exercise categories.
   Reveal Solution Discussion Select an answer first
2. 7expert · hard

   A healthcare AI system is being tested for diagnosing skin lesions. The system is non-deterministic due to stochastic inference, and the test team has a limited number of dermatologists available. They need to validate the system's output for each test image, but the dermatologists cannot review all images. The team is considering using a human oracle on a subset and a statistical oracle on the full set. Which approach best balances accuracy, cost, and coverage?

   • AUse a statistical oracle on all images and only involve dermatologists when the statistical oracle flags an anomaly.
   • BUse a heuristic oracle that flags images where the AI's confidence is below a threshold, and have dermatologists review only those.
   • CUse a model-based oracle that compares the AI's output to a previously approved version of the same model.
   • DHave dermatologists review all images, accepting the high cost and time.
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A medical imaging startup is developing an AI system that detects rare retinal diseases from fundus photographs. The team has a small set of images labeled by a single ophthalmologist, but the disease is so rare that they cannot obtain a large, diverse labeled dataset. They need to validate the system's output for each test image before release. Which oracle approach is most appropriate for this scenario?

   • AUse a statistical oracle that compares the model's confidence scores against a known distribution of disease prevalence.
   • BUse a heuristic oracle that flags any image where the AI's confidence is below 90% as a potential error.
   • CUse a model-based oracle by training a surrogate model on the same small labeled set to predict expected diagnoses.
   • DUse a human oracle, having the same ophthalmologist review each test image and compare their diagnosis to the AI's output.
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   Which advantage do human experts offer as test oracles for AI systems?

   • APerfect consistency across all test cases
   • BUnlimited scalability
   • CZero cost for large datasets
   • DAbility to judge complex or subjective outputs
   Reveal Solution Discussion Select an answer first
6. 10expert · hard · select all that apply

   A team is testing an AI system that generates realistic images of human faces. The system is generative and produces new images each run. The team needs to validate the quality of the generated images. Which oracle types are appropriate for this scenario? Select all that apply.

   • AHuman oracle, where human evaluators rate the realism of the images.
   • BExact-match oracle, comparing generated images to a fixed set of expected images.
   • CModel-based oracle, using a pre-trained classifier to detect if images look like real faces.
   • DStatistical oracle, comparing the distribution of generated images to that of real images.
   • EHeuristic oracle, checking that images meet basic constraints like resolution and color distribution.
   Check answers Discussion Select an answer first