A/B Testing CT-AI Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A company tests a new AI-based email subject line generator. The treatment group shows a 0.5% higher open rate, but the p-value is 0.08. What is the most appropriate decision?

   • ARoll out the new generator because a 0.5% improvement is practically meaningful.
   • BRun the test longer without recalculating the sample size.
   • CConclude that the new generator is worse than the old one.
   • DDo not roll out the new generator because the result is not statistically significant at the 0.05 level.
   Reveal Solution Discussion Select an answer first
2. 2expert · hard

   A social media platform tests a new AI-based content ranking algorithm. They use a 'holdout' group of users who never see the new algorithm. However, they notice that users in the treatment group interact with each other, potentially spreading the effect. What is the most appropriate way to handle this network effect?

   • AUse cluster randomization by social network communities to reduce spillover effects.
   • BRemove all users who have friends in the other group.
   • CUse a larger sample size to overcome the network effect.
   • DIgnore the network effect because it is usually small.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A streaming service tests a new AI-based content recommendation model. The product manager wants to measure user engagement. Which metric is most appropriate as the primary metric for this A/B test?

   • AServer response time of the recommendation API
   • BAverage watch time per user per week
   • CPercentage of users who rate a movie after watching
   • DNumber of new user sign-ups during the test
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   When interpreting A/B test results, why is it important to consider both statistical significance and practical significance?

   • APractical significance is more important than statistical significance, so the latter can be ignored.
   • BA result can be statistically significant but too small to matter for the business, or practically significant but not statistically significant due to insufficient sample size.
   • CStatistical significance is only relevant for large sample sizes, while practical significance is only relevant for small sample sizes.
   • DThey are the same thing and only one needs to be considered.
   Reveal Solution Discussion Select an answer first
6. 5expert · hard

   A team runs an A/B test for a new AI-based recommendation system. They collect data from 5,000 users per group. The p-value is 0.06, and the effect size is 0.5%. The team decides to collect more data until the p-value becomes significant. What is the main problem with this approach?

   • AIt makes the effect size larger.
   • BIt increases the risk of a false positive due to optional stopping.
   • CIt makes the treatment group larger than the control group.
   • DIt decreases the statistical power of the test.
   Reveal Solution Discussion Select an answer first