A/B Testing CT-AI Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   What is a common pitfall in A/B testing of AI-based systems?

   • APeeking at the results before the test has reached the planned sample size and stopping early based on a significant p-value.
   • BRandomly assigning users to control and treatment groups.
   • CPre-specifying the primary metric before the test begins.
   • DUsing a large sample size to ensure high statistical power.
   Reveal Solution Discussion Select an answer first
2. 7expert · hard

   A ride-sharing company tests a new AI-based surge pricing algorithm. They randomize users by city, with some cities in the treatment group and others in the control group. The treatment cities show a 5% increase in revenue, but the team is concerned about confounding. What is the most significant threat to the validity of this test?

   • AThe surge pricing algorithm is too complex to test with A/B testing.
   • BRevenue is not a valid metric for this test.
   • CCity-level differences (e.g., demographics, traffic patterns) may confound the results.
   • DThe sample size is too small because only a few cities are in each group.
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   Why can A/B testing be particularly challenging for AI-based systems compared to traditional software?

   • AAI systems are always deterministic, making it impossible to measure variance.
   • BAI systems do not produce measurable outcomes, so there is nothing to compare.
   • CAI systems can learn and adapt over time, so user behavior and system behavior may change during the test, potentially violating the assumption of a stable treatment effect.
   • DA/B testing is not applicable to AI systems because they cannot be split into control and treatment groups.
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   An e-commerce company runs an A/B test on a new product recommendation widget. They notice that users in the treatment group are more likely to be on mobile devices. What is the most appropriate action to mitigate this confounding variable?

   • AIgnore the imbalance because randomization should have balanced all variables.
   • BRemove all mobile users from the analysis.
   • CStratify the randomization by device type to ensure balanced groups across mobile and desktop users.
   • DSwitch the assignment of users to the opposite group after the test.
   Reveal Solution Discussion Select an answer first
6. 10foundation · easy

   In A/B testing, what does 'statistical significance' refer to?

   • AThe likelihood that the observed difference between groups is not due to random chance.
   • BThe practical importance of the observed difference.
   • CThe number of metrics that show a difference.
   • DThe size of the sample used in the test.
   Reveal Solution Discussion Select an answer first