Design A/B Tests CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   What is the primary purpose of an A/B test in a data science context?

   • ATo ensure that a model produces identical outputs for identical inputs
   • BTo replace the need for cross-validation when evaluating a predictive model
   • CTo compare two versions of a model or system to determine which performs better on a defined metric
   • DTo measure the absolute accuracy of a model on a fixed historical dataset
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   What is the 'peeking' problem in A/B testing?

   • AIncluding users who did not complete the test in the analysis
   • BLooking at the results before the test has reached its planned sample size and stopping early based on interim results
   • CUsing a small sample size that does not represent the population
   • DUsing a one-tailed test when a two-tailed test is appropriate
   Reveal Solution Discussion Select an answer first
4. 18expert · hard

   A company runs an A/B test on a new feature for its mobile app. The overall results show a statistically significant increase in user engagement. However, when the results are broken down by operating system (iOS vs. Android), the increase is only seen in iOS users, while Android users show a slight decrease. Which of the following is the most likely explanation?

   • AThe p-value was not adjusted for multiple testing, so the subgroup result is likely a false positive.
   • BSimpson's paradox: the overall result may be driven by the disproportionate size of the iOS group, masking the negative effect on Android.
   • CThe sample size was too small for the Android group, making the result unreliable.
   • DThe test was not randomized, so the groups were not comparable.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A team wants to test whether a new recommendation algorithm increases the click-through rate (CTR) compared to the current algorithm. They set up the following hypotheses: H0: CTR_new = CTR_old; H1: CTR_new ≠ CTR_old. They plan to run the test for two weeks. Which of the following is a valid concern about this approach?

   • AThe alternative hypothesis is two-sided, which is appropriate if the team is open to the new algorithm being either better or worse.
   • BThe null hypothesis should be that CTR_new is greater than CTR_old.
   • CThe test will require a larger sample size than a one-sided test, but it is still a valid approach.
   • DThe hypotheses are not correctly formulated because the null should be that CTR_new is less than or equal to CTR_old.
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A news website wants to test a new article layout. The primary goal is to increase the number of articles read per session. They also want to ensure the new layout does not increase page load time, which could hurt user experience. Which of the following is the best set of metrics for this test?

   • APrimary: time on page. Guardrail: number of sessions.
   • BPrimary: articles read per session. Guardrail: page load time.
   • CPrimary: page load time. Guardrail: articles read per session.
   • DPrimary: click-through rate on article links. Guardrail: bounce rate.
   Reveal Solution Discussion Select an answer first