A/B Testing CT-AI Practice Questions (Page 5)

Questions 21–25

1. 21expert · hard

   A team runs an A/B test for a new AI-based ad targeting model. After collecting the planned sample size, they find a p-value of 0.049. However, they also ran a secondary analysis on a different metric and found a p-value of 0.03. How should they interpret these results?

   • ABoth results are significant, so the new model is definitely better.
   • BThe primary metric is statistically significant, but the secondary analysis should be considered exploratory.
   • CThe team should ignore the primary metric and focus on the secondary metric because it has a lower p-value.
   • DThe test is invalid because the p-value is too close to 0.05.
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   After running an A/B test for a new AI chatbot, the p-value is 0.04. The team wants to conclude that the chatbot is better than the old one. What is the most appropriate interpretation of this result?

   • AThere is a 4% chance that the null hypothesis is true.
   • BThe observed difference is statistically significant at the 0.05 level, but practical significance should also be considered.
   • CThe new chatbot is guaranteed to perform better in all future scenarios.
   • DThe test is invalid because the p-value is not below 0.01.
   Reveal Solution Discussion Select an answer first
4. 23expert · hard

   A team is testing a new AI-based recommendation algorithm. They decide to assign users to groups based on the last digit of their user ID (odd/even). They believe this is random. What is the most likely problem with this approach?

   • AUser IDs may not be randomly assigned, leading to systematic differences between groups.
   • BThe test will have low statistical power.
   • CThe treatment group will be biased towards new users.
   • DThe groups will not be equal in size.
   Reveal Solution Discussion Select an answer first
5. 24expert · hard

   A team is designing an A/B test for a new AI-based recommendation system. They have a limited budget and can only include 10,000 users in the test. They want to detect a 1% increase in click-through rate (CTR) from a baseline of 5%. Using a power analysis, they find they need 15,000 users. What is the most appropriate course of action?

   • AUse a one-tailed test instead of a two-tailed test to reduce the sample size.
   • BReduce the significance level to 0.01 to make the test more rigorous.
   • CIncrease the minimum detectable effect size to 1.5% to fit the budget.
   • DRun the test with 10,000 users and accept the lower power.
   Reveal Solution Discussion Select an answer first
6. 25foundation · easy

   Which of the following is the most important characteristic of a metric used in an A/B test?

   • AIt must be aligned with the business goal and sensitive enough to detect changes caused by the treatment.
   • BIt must be the same for all users regardless of their behavior.
   • CIt must be a single number that summarizes the overall performance of the system.
   • DIt must be easy to compute and require minimal data collection.
   Reveal Solution Discussion Select an answer first