A data science team is designing an A/B test for a new feature. They plan to analyze the results after every 1,000 users complete the test and stop as soon as the p-value is below 0.05. Which of the following is the most appropriate way to address the issue with this approach?

Use a larger significance level, such as 0.10, to account for the multiple looks.

Pre-specify a fixed sample size and analyze the data only once at the end.

Run the test for a longer duration to ensure the effect is stable.

Use a one-tailed test to increase the chance of finding significance.

A data scientist runs an A/B test with 5,000 users per group. The mean time on site for the control is 120 seconds (sd=30) and for the treatment is 125 seconds (sd=30). A two-sample t-test yields a p-value of 0.03. Which of the following is the most appropriate conclusion?

The difference is statistically significant at the 0.05 level, but the practical significance depends on the business context.

The difference is not statistically significant because the p-value is too close to 0.05.

The test is invalid because the sample size is too small.

The treatment is definitely better because the mean is higher.

A data science team at a retail company wants to test whether a new recommendation algorithm increases the average order value (AOV) compared to the current algorithm. They plan to run an A/B test for one week and want to detect a minimum lift of $5 in AOV. Based on historical data, the standard deviation of AOV is $40. They set significance level α=0.05 and power=0.80. Which sample size per group is most appropriate?

Approximately 64 users per group

Approximately 1,006 users per group

Approximately 2,012 users per group

Approximately 503 users per group

A data scientist is planning an A/B test to compare two versions of a sign-up form. The current form has a 5% conversion rate. They want to detect a 1 percentage point increase (to 6%) with 80% power and a 5% significance level. Which of the following is the most appropriate next step?

Set the significance level to 0.10 to reduce the required sample size.

Use a one-sample t-test to compare the conversion rates.

Run the test for a fixed duration of one month regardless of sample size.

Calculate the required sample size using a two-proportion test formula.

What is the role of a control group in an A/B test?

To increase the sample size and thus the statistical power

To test multiple variations simultaneously

To receive the new version and measure its absolute performance

To provide a baseline for comparison, allowing the effect of the change to be isolated

CertNexusCertified Data Science Practitioner (CDSP)

Domain 5Objective 1

Design A/B Tests CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 3)

Part of the 5.0 Testing models domain, which accounts for 4-7% of the CERTIFIED-DATA-SCIENCE-PRACTITIONER exam.

27questions here

6free pages

8concepts

4-7%of the exam

Questions 11–15

11expert · hard
A data science team is designing an A/B test for a new feature. They plan to analyze the results after every 1,000 users complete the test and stop as soon as the p-value is below 0.05. Which of the following is the most appropriate way to address the issue with this approach?
Select an answer first
12application · medium
A data scientist runs an A/B test with 5,000 users per group. The mean time on site for the control is 120 seconds (sd=30) and for the treatment is 125 seconds (sd=30). A two-sample t-test yields a p-value of 0.03. Which of the following is the most appropriate conclusion?
Select an answer first
13application · medium
A data science team at a retail company wants to test whether a new recommendation algorithm increases the average order value (AOV) compared to the current algorithm. They plan to run an A/B test for one week and want to detect a minimum lift of $5 in AOV. Based on historical data, the standard deviation of AOV is $40. They set significance level α=0.05 and power=0.80. Which sample size per group is most appropriate?
Select an answer first
14application · medium
A data scientist is planning an A/B test to compare two versions of a sign-up form. The current form has a 5% conversion rate. They want to detect a 1 percentage point increase (to 6%) with 80% power and a 5% significance level. Which of the following is the most appropriate next step?
Select an answer first
15foundation · easy
What is the role of a control group in an A/B test?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CERTIFIED-DATA-SCIENCE-PRACTITIONER

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by CertNexus. “CERTIFIED-DATA-SCIENCE-PRACTITIONER” is a trademark of its owner, used for identification only.