Design A/B Tests CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   Which of the following is an example of a good primary success metric for an A/B test on a new recommendation algorithm?

   • AThe click-through rate on recommended items
   • BThe number of engineers on the team
   • CThe total number of users in the test
   • DThe color of the recommendation button
   Reveal Solution Discussion Select an answer first
2. 7foundation · easy

   Which of the following factors is NOT required to calculate the required sample size for an A/B test?

   • ASignificance level (alpha)
   • BDesired statistical power
   • CThe color scheme of the user interface
   • DThe expected effect size
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   What is Simpson's paradox in the context of A/B testing?

   • AA trend that appears in different groups of data but disappears or reverses when the groups are combined
   • BThe increase in false positives when multiple metrics are tested
   • CThe decrease in statistical power when the sample size is too small
   • DThe tendency for users to behave differently when they know they are in a test
   Reveal Solution Discussion Select an answer first
5. 9expert · hard

   An e-commerce company runs an A/B test on a new checkout flow. The 95% confidence interval for the difference in conversion rate (new - old) is [-0.2%, +1.8%]. The company's business team says they would only consider the new flow worthwhile if it increases conversion by at least 1%. Which of the following is the most appropriate interpretation?

   • AThe test is inconclusive because the confidence interval is too wide.
   • BThe new flow is statistically significant because the confidence interval does not include zero.
   • CThe result is not practically significant because the confidence interval includes values below the 1% threshold, so the true effect could be negligible.
   • DThe new flow is practically significant because the upper bound of the confidence interval exceeds 1%.
   Reveal Solution Discussion Select an answer first
6. 10foundation · easy

   What does a 95% confidence interval for the difference in conversion rates between two versions tell you?

   • AIf the interval does not contain zero, the difference is statistically significant at the 5% level
   • BThe sample size is sufficient for the test
   • CThere is a 95% probability that the true difference lies within the interval
   • DThe true difference is guaranteed to be within the interval
   Reveal Solution Discussion Select an answer first