Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems CT-AI Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   When evaluating the reliability of a probabilistic AI system, what should be considered?

   • AThe system's speed of processing inputs.
   • BThe system's ability to run on different hardware.
   • CThe system's ability to produce the same output every time.
   • DThe consistency of the system's performance across multiple runs.
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A team is testing a recommendation system that uses stochastic gradient descent with random batch sampling during training. When they run the same test suite twice against the same trained model, they get different recommendation lists for the same user profile. The test lead insists this is a defect and wants the testers to file a bug. What is the most appropriate first response?

   • AExplain that the model is probabilistic and the variation may be expected, then investigate whether the variation is within an acceptable range before filing a bug.
   • BFile the bug immediately because any difference in output between two runs of the same test indicates a flaky test that must be fixed.
   • CRerun the test suite multiple times and only file a bug if the output differs in more than half of the runs.
   • DSet a fixed random seed in the test environment and rerun the tests to confirm the outputs become identical before filing a bug.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A team is testing a fraud-detection model that returns a probability score. They run 100 test cases and observe that the model's accuracy is 92%. They want to know if the model is significantly better than a baseline accuracy of 90%. Which statistical approach should they use?

   • AReport the 92% accuracy as the model's true performance since it is based on 100 test cases.
   • BIncrease the test set to 1000 cases and then compare the accuracy directly without any statistical test.
   • CPerform a hypothesis test comparing the observed accuracy to the baseline, using a significance level such as 0.05.
   • DUse a confidence interval for the baseline accuracy and check if 92% falls inside it.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   What is a key characteristic of a test oracle for a non-deterministic AI system?

   • AIt must be based on the system's internal implementation details.
   • BIt must define an acceptable range or set of outputs for each input.
   • CIt must specify a single exact expected output for each input.
   • DIt must be the same for all test cases.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   Why are traditional testing approaches often insufficient for probabilistic AI systems?

   • ATraditional tests are too slow to run on AI systems.
   • BTraditional tests are designed for hardware, not software.
   • CTraditional tests assume deterministic behavior, so they cannot handle output variability.
   • DTraditional tests require a large amount of data, which AI systems do not have.
   Reveal Solution Discussion Select an answer first