Back-to-Back Testing CT-AI Practice Questions (Page 5)

Questions 21–25

1. 21foundation · easy

   What does a discrepancy between outputs of two versions indicate in back-to-back testing?

   • AThat the system is non-deterministic and cannot be tested
   • BThat both versions are definitely correct
   • CThat the test inputs were invalid
   • DThat at least one version may have a defect
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A team is back-to-back testing a natural language generation model. The model is non-deterministic: running the same input twice can produce slightly different wording. They run both the old and new versions once per input and find that 30% of outputs differ. What is the most appropriate next step?

   • ATreat all 30% of differences as defects and roll back to the old version.
   • BSet a random seed on both models to force identical outputs, then compare them exactly.
   • CRun each version multiple times per input and compare the distributions of outputs using a statistical similarity measure.
   • DOnly compare outputs that are exactly identical and ignore all others.
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   In which scenario is back-to-back testing particularly useful for an AI system?

   • AWhen testing the system's response time under load
   • BWhen verifying that the system meets user interface design specifications
   • CWhen comparing a newly trained model to a previously deployed model
   • DWhen checking the system's ability to handle malicious inputs
   Reveal Solution Discussion Select an answer first
5. 24expert · hard

   A team is back-to-back testing a new version of a medical image segmentation model. The old and new models agree on 95% of pixels, but disagree on the remaining 5%. The team has a small labeled dataset for the disagreement regions. They find that the new model is correct on 70% of the disagreement regions, and the old model is correct on 30%. What is the most important next step before deploying the new model?

   • ARoll back to the old model because it is the known reference.
   • BInvestigate the 30% of disagreement regions where the old model was correct to understand if the new model introduces new types of errors.
   • CRetrain the new model on all disagreement regions using the labels.
   • DDeploy the new model because it is correct on 70% of the disagreement regions.
   Reveal Solution Discussion Select an answer first
6. 25application · medium

   A team is back-to-back testing a reinforcement learning agent that controls a simulated robot. The agent's behavior is stochastic, so the same starting state can lead to different trajectories. The team wants to compare a new policy against the current one. What is the most appropriate way to handle the stochasticity?

   • ACompare the policies only on the final reward, ignoring intermediate steps.
   • BRun each policy once per starting state and compare the exact trajectories.
   • CRun multiple episodes from the same set of starting states for each policy and compare the distribution of cumulative rewards.
   • DDisable randomness in the environment so both policies produce deterministic trajectories.
   Reveal Solution Discussion Select an answer first