Experimental Design CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A data scientist is analyzing the results of an A/B test. They notice that the treatment group has a slightly higher average order value, but the difference is not statistically significant. They decide to segment the data by user location and find that in one specific city, the difference is significant. What is the most appropriate interpretation?

   • AThe overall result is not significant, so the apparent effect in one city is likely due to chance from multiple testing.
   • BThe treatment works in that city, so it should be rolled out there.
   • CThe experiment should be rerun with only users from that city to confirm the effect.
   • DThe treatment is effective overall, but the effect is diluted by other cities.
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A data scientist is planning an experiment to test a new feature. They have defined the null and alternative hypotheses, selected the primary metric, and determined the sample size. They are now ready to run the experiment. What is the most important remaining step before launching?

   • ACollect additional data from non-random sources to increase the sample size.
   • BChoose the statistical test after seeing the results to ensure significance.
   • CRun the experiment as quickly as possible to collect data.
   • DPre-register the analysis plan, including the statistical test and any subgroup analyses.
   Reveal Solution Discussion Select an answer first
4. 23expert · hard

   A data scientist is designing an experiment to test a new recommendation algorithm. They are concerned that the time of day users interact with the app might confound the results. They decide to randomly assign users to treatment or control within each hour of the day. However, they notice that the number of users varies greatly by hour, with some hours having very few users. What is the most appropriate way to handle this?

   • AUse simple randomization without blocking, because the hour effect is likely small.
   • BExclude hours with low traffic from the experiment to simplify the design.
   • CUse a factorial design with hour as a factor to test the interaction.
   • DCombine hours with low traffic into larger blocks to ensure adequate sample size in each block.
   Reveal Solution Discussion Select an answer first
5. 24expert · hard

   A data scientist is analyzing the results of a factorial experiment with two factors: price (low vs. high) and ad type (image vs. video). The ANOVA results show a significant interaction effect between price and ad type. What is the most appropriate next step?

   • AInterpret the main effects of price and ad type separately.
   • BRerun the experiment without the interaction term to simplify the analysis.
   • CExamine the simple effects of each factor at each level of the other factor.
   • DConclude that both factors are significant and recommend the best combination.
   Reveal Solution Discussion Select an answer first
6. 25application · medium

   A company is running an experiment to test a new loyalty program. They are concerned that the effect might be different for high-spending vs. low-spending customers. They decide to divide customers into high-spenders and low-spenders, then randomly assign each group to either the new program or the current program. This approach is an example of:

   • AQuasi-experimental design because randomization is not used.
   • BBlocked design with spending level as the blocking variable.
   • CRandomized controlled trial without any blocking.
   • DFactorial design with spending level as a factor.
   Reveal Solution Discussion Select an answer first