Design A/B Tests CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   Which statistical test is most appropriate for comparing the means of two independent groups in an A/B test?

   • AANOVA
   • BChi-square test
   • CTwo-sample t-test
   • DCorrelation test
   Reveal Solution Discussion Select an answer first
2. 2foundation · easy

   If you increase the desired statistical power of an A/B test (while keeping other factors constant), what happens to the required sample size?

   • AIt increases
   • BIt becomes unpredictable
   • CIt stays the same
   • DIt decreases
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A social media platform wants to test a new algorithm that aims to increase user engagement. They define the primary metric as 'daily active users' (DAU). However, they are concerned that the new algorithm might increase DAU by showing more addictive content, which could harm long-term user well-being. Which of the following is the best approach to address this concern?

   • AChange the primary metric to 'user satisfaction score' instead of DAU.
   • BUse a longer test duration to ensure the effect on DAU is stable.
   • CAdd a guardrail metric such as 'user-reported negative feedback' or 'time spent per session' to monitor potential harm.
   • DRun the test only on a small subset of users to minimize potential harm.
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A team is considering using an A/B test to compare two different recommendation models. They have a large user base and can randomly assign users to each model. Which of the following is the primary purpose of using a control group in this A/B test?

   • ATo ensure that the treatment group receives the new model without any bias.
   • BTo allow the team to test multiple versions of the model simultaneously.
   • CTo increase the sample size and improve the statistical power of the test.
   • DTo provide a baseline for comparison, allowing the team to measure the effect of the new model relative to the current one.
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A data scientist runs an A/B test comparing two versions of a landing page. The conversion rate for the control is 10% and for the treatment is 12% (n=1,000 per group). A chi-square test yields a p-value of 0.14. Which of the following is the most appropriate conclusion?

   • AThere is not enough evidence to conclude that the treatment page has a different conversion rate from the control at the 0.05 significance level.
   • BThe treatment page is significantly worse because the p-value is not low enough.
   • CThe test is underpowered, so the results are inconclusive and the test should be rerun with a larger sample size.
   • DThe treatment page is significantly better because the conversion rate is higher.
   Reveal Solution Discussion Select an answer first