Experimental Design CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   A data science team wants to compare two versions of a recommendation algorithm by randomly assigning users to see one version or the other and measuring click-through rate. Which experimental design does this describe?

   • AQuasi-experimental design
   • BRandomized controlled trial
   • CFactorial design
   • DA/B testing
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A data scientist runs an A/B test with 5,000 users per group. The conversion rate for the control is 8% and for the treatment is 9%. The p-value is 0.07. The business team is eager to launch the new feature because the 1% absolute increase is meaningful for revenue. What should the data scientist recommend?

   • ARun the experiment longer to increase the sample size and achieve statistical significance.
   • BLaunch the feature because the effect size is practically significant.
   • CConsider the trade-off between false positive and false negative risks, and possibly run a follow-up experiment with a larger sample size to confirm the effect.
   • DDo not launch the feature because the p-value is above 0.05.
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   A data science team is about to run an experiment to test a new recommendation algorithm. They have defined the null hypothesis (no difference in click-through rate) and the alternative hypothesis (new algorithm increases click-through rate). They have also selected the primary metric and determined the sample size. What is the next essential step in the experimental design process?

   • ACollect additional data from a non-random sample to increase the sample size.
   • BAnalyze the results using a t-test without specifying the analysis plan in advance.
   • CRun the experiment immediately to save time.
   • DDefine the randomization and control procedures, including how users are assigned to groups and how confounders are handled.
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A team is running an A/B test on a website. They randomly assign users to see either the current homepage or a new homepage. However, they later discover that the randomization was not truly random because users with certain browser types were more likely to be assigned to the new homepage due to a bug. What is the most appropriate way to handle this?

   • AIgnore the issue because the sample size is large enough to overcome the imbalance.
   • BContinue the experiment and adjust for browser type in the analysis.
   • CRerun the experiment with the same data but use a different statistical test.
   • DStop the experiment and fix the randomization bug before drawing any conclusions.
   Reveal Solution Discussion Select an answer first
6. 20foundation · easy

   Which of the following is a method to control for confounding variables in an experiment?

   • AUsing a larger dataset
   • BRandom assignment of subjects to groups
   • CUsing a more complex model
   • DRemoving outliers from the data
   Reveal Solution Discussion Select an answer first