Back-to-Back Testing CT-AI Practice Questions (Page 4)

Questions 16–20

1. 16expert · hard

   A team is back-to-back testing a time-series forecasting model. The output is a continuous value (e.g., predicted temperature). The old and new models differ by an average of 0.5 degrees Celsius. The team needs to decide if this difference is acceptable. Which approach is most appropriate?

   • AUse the average difference alone to decide if the models are equivalent.
   • BRequire the models to produce exactly the same output for all inputs.
   • CDefine a tolerance threshold based on the domain requirements (e.g., ±1 degree) and investigate cases where the difference exceeds the threshold.
   • DRun each model once and compare the outputs visually on a plot.
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A team is migrating a machine learning model from a Python framework to a Java framework for performance reasons. They want to ensure the Java implementation produces the same predictions as the Python one. They have a large set of historical inputs but no labels. What is the most appropriate testing strategy?

   • AManually label a subset of the historical inputs and evaluate both implementations on it.
   • BRun both implementations on the same historical inputs and compare their predictions.
   • CDeploy the Java implementation and compare its predictions with the Python implementation in production.
   • DTest the Java implementation with unit tests written from the Python implementation's documentation.
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   What is the first step in performing back-to-back testing?

   • AFixing any discrepancies found
   • BComparing the outputs of the versions
   • CDefining the expected outputs for each test case
   • DSelecting a set of test inputs to use with all versions
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A team has updated a credit-scoring model to comply with new regulations. The new model must produce the same decisions as the old model for most applicants, but with a different internal calculation. They have historical applicant data but no ground truth for 'correct' credit decisions. What is the most appropriate way to validate the new model?

   • ARun both models on the historical applicant data and compare their credit decisions.
   • BUse a third-party model to evaluate the new model's decisions.
   • CDeploy the new model and monitor for customer complaints.
   • DReview the new model's code to ensure it implements the new calculation correctly.
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A team has replaced a legacy image-classification model with a newer architecture that is 15% more accurate on the validation set. Before deploying, they want to confirm the new model has not introduced unexpected behavior changes on inputs the old model handled correctly. They have no ground-truth labels for their production data. Which approach best addresses their concern?

   • ARun both models on the same production inputs and compare their predictions, investigating cases where they disagree.
   • BRetrain the new model on the validation set until its accuracy matches the old model's accuracy.
   • CUse the old model's predictions as pseudo-labels to fine-tune the new model further.
   • DDeploy the new model and monitor its accuracy on live traffic for two weeks.
   Reveal Solution Discussion Select an answer first