Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems CT-AI Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A team is testing a natural language processing model that uses dropout during inference. A test that checks the model's output for a specific input fails intermittently. The testers have confirmed the model is correct when it passes. What is the most appropriate strategy to handle this flaky test?

   • ARerun the test multiple times and report a pass if it passes at least once.
   • BSet a fixed random seed for the dropout during inference.
   • CIgnore the test failure and rely on other tests that pass consistently.
   • DDisable dropout during inference in the test environment to make the model deterministic.
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A team is testing a speech-recognition system that transcribes audio. Due to environmental noise, the same audio clip can produce slightly different transcriptions across runs. The product owner wants a test oracle to determine if a transcription is correct. Which oracle design is most appropriate for this probabilistic system?

   • ARun the test once and manually review the transcription to decide if it is correct.
   • BAccept a transcription as correct if its word error rate (WER) against a reference transcript is below a defined threshold, such as 10%.
   • CRequire the transcription to exactly match the reference transcript for the test to pass.
   • DUse the most frequent transcription from multiple runs as the expected output and compare against it.
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A team is evaluating a chatbot that generates responses using a language model with a temperature parameter. They want to assess the chatbot's reliability. Which approach best evaluates reliability in the presence of non-determinism?

   • ASet the temperature to zero and test the chatbot with a single run per prompt.
   • BCompare the chatbot's responses to a set of pre-defined expected responses.
   • CSend the same set of prompts to the chatbot multiple times and measure the consistency of the responses against a rubric.
   • DSend each prompt once and check if the response is grammatically correct.
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   In hypothesis testing for a probabilistic AI system, what does a p-value represent?

   • AThe probability that the null hypothesis is true.
   • BThe probability that the alternative hypothesis is false.
   • CThe probability of obtaining the observed results, or more extreme, assuming the null hypothesis is true.
   • DThe probability that the test will fail on the next run.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A team is testing a spam classifier that outputs a probability of being spam. They have a test set of 500 emails and want to verify that the classifier's performance is stable across multiple runs. The classifier uses a stochastic training algorithm, so each run produces a slightly different model. What is the best way to evaluate the stability of the classifier's performance?

   • ATrain the model multiple times and compute the mean and standard deviation of the accuracy across the runs.
   • BSplit the test set into multiple subsets and test the model on each subset.
   • CTrain the model once and use the accuracy from that single run as the performance estimate.
   • DUse a fixed random seed for training and test the model once.
   Reveal Solution Discussion Select an answer first