Design A/B Tests CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A team wants to test a new pricing page. They plan to assign users to the new page based on their user ID (even IDs get the new page, odd IDs get the old page). Which of the following is the most important reason this assignment is not ideal?

   • AIt violates the assumption of independence of observations.
   • BIt does not allow for a control group.
   • CIt may not produce truly random groups if user ID is correlated with user characteristics.
   • DIt will result in unequal sample sizes.
   Reveal Solution Discussion Select an answer first
2. 22foundation · easy

   Why is random assignment of users to groups important in an A/B test?

   • AIt allows the test to be run without a control group
   • BIt guarantees that the results will be statistically significant
   • CIt helps to eliminate selection bias and balance known and unknown confounding variables across groups
   • DIt ensures the groups are identical in every possible way
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   What is the purpose of a guardrail metric in an A/B test?

   • ATo monitor for unintended negative effects on other important metrics
   • BTo replace the primary success metric
   • CTo determine the sample size for the test
   • DTo ensure the test is statistically significant
   Reveal Solution Discussion Select an answer first
5. 24foundation · easy

   In an A/B test, what does the 'control' group typically receive?

   • ANo version of the system at all
   • BThe current or baseline version of the system
   • CBoth versions simultaneously to measure interaction effects
   • DThe new or experimental version of the system
   Reveal Solution Discussion Select an answer first
6. 25expert · hard

   A product team runs an A/B test for a new checkout flow. They check the p-value every day and plan to stop the test as soon as it drops below 0.05. After three days, the p-value is 0.04, so they stop the test and declare the new flow better. Which of the following best describes the main problem with this approach?

   • APeeking at the data multiple times inflates the probability of a false positive, so the observed p-value is not a reliable indicator of significance.
   • BThe sample size was too small, so the test was underpowered.
   • CThe p-value should be adjusted for multiple comparisons across the two versions.
   • DThe test should have used a one-tailed test instead of a two-tailed test to increase power.
   Reveal Solution Discussion Select an answer first