Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems CT-AI Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A test team has evaluated a predictive model and found that its accuracy on a test set is 88%, with a 95% confidence interval of [85%, 91%]. The product owner asks whether the model is good enough to deploy. What is the most appropriate way to communicate this result?

   • AReport the accuracy along with the confidence interval and explain that the true accuracy is likely between 85% and 91%, and let the product owner decide based on business requirements.
   • BReport only the 88% accuracy as the model's performance, since the confidence interval is just a technical detail.
   • CState that the model is not reliable because the confidence interval is too wide.
   • DTell the product owner that the model is guaranteed to be at least 85% accurate in production.
   Reveal Solution Discussion Select an answer first
2. 2foundation · easy

   Which statement best distinguishes a probabilistic AI system from a deterministic one?

   • AA probabilistic system is one that has been trained on probabilistic data, while a deterministic system is trained on deterministic data.
   • BA probabilistic system can produce different outputs for the same input across runs, while a deterministic system always produces the same output for the same input.
   • CA probabilistic system is always slower than a deterministic system because it must compute probabilities.
   • DA probabilistic system always produces the same output for the same input, while a deterministic system may vary.
   Reveal Solution Discussion Select an answer first
4. 3foundation · easy

   What is a common approach to handle flaky tests in a continuous integration (CI) pipeline?

   • AIncrease the test timeout to give the system more time.
   • BAutomatically rerun the failed test to see if it passes on retry.
   • CMark the test as 'expected to fail' and ignore it.
   • DDelete the flaky test from the test suite.
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A team is testing an image-classification model that uses random data augmentation during inference. A test that checks the model's accuracy on a fixed set of images sometimes passes and sometimes fails. The testers suspect the random augmentation is causing the flakiness. What is the most effective way to make the test reproducible?

   • ARerun the test until it passes and report the passing result.
   • BIncrease the number of test images so the accuracy is more stable.
   • CRemove the random augmentation from the model during testing.
   • DSet a fixed random seed for the augmentation process in the test environment.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   Which strategy can help mitigate flaky tests caused by randomness in an AI system?

   • AIgnoring test failures that occur intermittently.
   • BSetting a fixed random seed for all random operations.
   • CIncreasing the number of test cases to cover more scenarios.
   • DRunning tests only once to avoid variability.
   Reveal Solution Discussion Select an answer first