Design A/B Tests CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 6)

Questions 26–27

1. 26application · medium

   A mobile app team wants to test a new onboarding flow. They plan to assign users to the new flow based on the day of the week they first open the app (e.g., Monday/Wednesday/Friday get the new flow, others get the old flow). Which of the following is the most important reason this assignment method is problematic?

   • AIt makes the test results less generalizable to the entire user base.
   • BIt violates the independence assumption of the t-test.
   • CIt requires a larger sample size to achieve the same power.
   • DIt does not ensure random assignment, so the groups may differ in characteristics that affect the success metric.
   Reveal Solution Discussion Select an answer first
2. 27foundation · easy

   In an A/B test where the success metric is a binary outcome (e.g., clicked or not), which statistical test is commonly used?

   • AANOVA
   • BRegression analysis
   • CChi-square test
   • DPaired t-test
   Reveal Solution Discussion Select an answer first