Which of the following is a key component of experimental design in the context of data science model testing?

Deploying the model to production immediately after training

Defining the hypothesis and selecting the response variable

Ensuring the model has high accuracy on the training data

Choosing the programming language for implementation

After running an A/B test comparing two model versions, what statistical technique is commonly used to determine if the observed difference in performance metrics is statistically significant?

Hypothesis testing (e.g., t-test or chi-square test)

What is the primary role of randomization in experimental design?

To ensure the model is computationally efficient

To reduce bias by ensuring each subject has an equal chance of being assigned to any group

To guarantee that the results are statistically significant

To increase the sample size without additional cost

When planning an experiment to test a new feature in a model, what is the first step a data scientist should take?

Collect as much data as possible

Define a clear, testable hypothesis

A data science team is designing an experiment to test whether a new recommendation algorithm increases average user session time compared to the current algorithm. They have a large user base and want to detect a small but meaningful effect. They are concerned about confounding variables such as user device type and time of day. Which approach best addresses both the need to detect a small effect and control for these confounders?

Increase the sample size only, because a large sample automatically controls for all confounding variables.

Run a factorial design with all combinations of device type and time-of-day to test every possible interaction effect.

Run a randomized controlled trial with a large sample size and stratify users by device type and time-of-day segment before random assignment.

Use a simple random sample of users without any stratification to keep the experiment simple and unbiased.

CertNexusCertified Data Science Practitioner (CDSP)

Domain 5Objective 2

Experimental Design CERTIFIED-DATA-SCIENCE-PRACTITIONER Practice Questions (Page 3)

Part of the 5.0 Testing models domain, which accounts for 4-7% of the CERTIFIED-DATA-SCIENCE-PRACTITIONER exam.

26questions here

6free pages

6concepts

4-7%of the exam

Questions 11–15

11foundation · easy
Which of the following is a key component of experimental design in the context of data science model testing?
Select an answer first
12foundation · easy
After running an A/B test comparing two model versions, what statistical technique is commonly used to determine if the observed difference in performance metrics is statistically significant?
Select an answer first
13foundation · easy
What is the primary role of randomization in experimental design?
Select an answer first
14foundation · easy
When planning an experiment to test a new feature in a model, what is the first step a data scientist should take?
Select an answer first
15application · medium
A data science team is designing an experiment to test whether a new recommendation algorithm increases average user session time compared to the current algorithm. They have a large user base and want to detect a small but meaningful effect. They are concerned about confounding variables such as user device type and time of day. Which approach best addresses both the need to detect a small effect and control for these confounders?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CERTIFIED-DATA-SCIENCE-PRACTITIONER

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by CertNexus. “CERTIFIED-DATA-SCIENCE-PRACTITIONER” is a trademark of its owner, used for identification only.