A team is running an A/B test on a new AI-based search ranking algorithm. They check the results every day and decide to stop the test as soon as the p-value drops below 0.05. What is the main pitfall of this approach?

It causes the treatment group to be larger than the control group.

It decreases the statistical power of the test.

It makes the randomization invalid.

It increases the risk of a false positive due to multiple testing.

What is the primary factor that determines the required sample size for an A/B test?

The number of metrics being measured in the test.

The total number of users available in the system.

The desired statistical power and the minimum effect size you want to detect.

The duration of the test in days.

A team is designing an A/B test for a new fraud detection model. They want to detect a 1% improvement in detection rate with 80% power and a 5% significance level. What is the primary factor that determines the required sample size?

The baseline conversion rate and the minimum detectable effect size

The number of features in the AI model

The type of machine learning algorithm used

The duration of the test in days

What does a p-value of 0.03 indicate in the context of an A/B test?

The treatment group performed 3% better than the control group.

There is a 97% probability that the treatment is effective.

The null hypothesis is definitely false.

There is a 3% probability that the observed difference is due to random chance, assuming the null hypothesis is true.

What is the primary purpose of A/B testing in the context of evaluating an AI-based system?

To ensure that the AI system's decisions are explainable and interpretable to end users.

To test the system's ability to handle a large number of concurrent users by simulating high traffic loads.

To verify that the AI system's outputs are correct against a set of labeled test cases.

To compare two versions of a system by randomly assigning users to each version and measuring a predefined outcome to determine which performs better.

Certified Tester AI Testing

Domain 9Objective 5

A/B Testing CT-AI Practice Questions (Page 4)

Part of the Domain 9: Methods and Techniques for the Testing of AI-Based Systems domain, which makes up ~13% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~3–5 in this domain), expect 1–1 from this objective — we provide 26 practice questions to prepare you well beyond it. (estimate)

26questions here

6free pages

9concepts

Questions 16–20

16application · medium
A team is running an A/B test on a new AI-based search ranking algorithm. They check the results every day and decide to stop the test as soon as the p-value drops below 0.05. What is the main pitfall of this approach?
Select an answer first
17foundation · easy
What is the primary factor that determines the required sample size for an A/B test?
Select an answer first
18application · medium
A team is designing an A/B test for a new fraud detection model. They want to detect a 1% improvement in detection rate with 80% power and a 5% significance level. What is the primary factor that determines the required sample size?
Select an answer first
19foundation · easy
What does a p-value of 0.03 indicate in the context of an A/B test?
Select an answer first
20foundation · easy
What is the primary purpose of A/B testing in the context of evaluating an AI-based system?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-AI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-AI” is a trademark of its owner, used for identification only.