After an A/B test, you find that the treatment group has a higher conversion rate than the control group, and the p-value is 0.04. What is the most appropriate interpretation?

The treatment is guaranteed to improve conversion rate for all users.

There is statistically significant evidence that the treatment improves conversion rate, assuming a significance level of 0.05.

The treatment has a 4% chance of being worse than the control.

The result is not meaningful because the p-value is not below 0.01.

Which of the following is an effective way to mitigate the impact of a known confounding variable in an A/B test?

Removing all users who exhibit the confounding variable from the test.

Stratified randomization, where users are first divided into subgroups based on the confounder and then randomly assigned within each subgroup.

Ignoring the confounder because randomization will automatically balance it.

Increasing the sample size to make the confounder less influential.

In an A/B test, what is the purpose of the control group?

To ensure that the test is double-blind.

To provide a baseline against which the treatment group's performance is compared.

To increase the sample size of the test.

To receive the new experimental version of the AI system.

In an A/B test of an AI-based recommendation system, what does the 'treatment group' typically receive?

The current production recommendation algorithm.

The new recommendation algorithm being evaluated.

No recommendations at all, to serve as a baseline.

A random selection of recommendations from both the old and new algorithms.

For an A/B test evaluating a new AI-powered search ranking algorithm, which metric would be most appropriate as a primary metric?

Click-through rate (CTR) on search results.

The total number of queries processed during the test.

The average age of users in the test.

The number of servers used to host the search service.

Certified Tester AI Testing

Domain 9Objective 5

A/B Testing CT-AI Practice Questions (Page 3)

Part of the Domain 9: Methods and Techniques for the Testing of AI-Based Systems domain, which makes up ~13% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~3–5 in this domain), expect 1–1 from this objective — we provide 26 practice questions to prepare you well beyond it. (estimate)

26questions here

6free pages

9concepts

Questions 11–15

11foundation · easy
After an A/B test, you find that the treatment group has a higher conversion rate than the control group, and the p-value is 0.04. What is the most appropriate interpretation?
Select an answer first
12foundation · easy
Which of the following is an effective way to mitigate the impact of a known confounding variable in an A/B test?
Select an answer first
13foundation · easy
In an A/B test, what is the purpose of the control group?
Select an answer first
14foundation · easy
In an A/B test of an AI-based recommendation system, what does the 'treatment group' typically receive?
Select an answer first
15foundation · easy
For an A/B test evaluating a new AI-powered search ranking algorithm, which metric would be most appropriate as a primary metric?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-AI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-AI” is a trademark of its owner, used for identification only.