A financial firm is testing an AI system that predicts daily stock price movements. The system is non-deterministic: running the same input twice can produce slightly different outputs due to stochastic elements. The test team needs an oracle to judge whether a prediction is acceptable. Which oracle approach is most suitable?

A human oracle where a trader manually evaluates each prediction against their intuition.

A statistical oracle that checks whether the distribution of predictions over many runs matches the expected distribution of price movements.

A heuristic oracle that flags predictions outside a predefined percentage range from the previous day's closing price.

A model-based oracle that compares each prediction to the output of a deterministic reference model.

A team is testing an AI system that predicts whether a loan applicant will default. The system outputs a probability of default. The team has historical data on actual defaults for a test set. They are evaluating two candidate oracles: Oracle A flags applicants with predicted probability > 0.5, and Oracle B flags applicants with predicted probability > 0.7. The team wants to minimize the risk of missing actual defaults (high recall) while keeping the number of false alarms manageable. Which oracle should they choose?

Neither oracle, because thresholds are arbitrary and cannot be used as oracles.

Oracle A, because it has higher recall and will catch more actual defaults.

Oracle B, because it has higher precision and will have fewer false alarms.

Both oracles, because they provide complementary information.

What does the precision of a test oracle measure?

The proportion of flagged issues that are actual faults

The speed at which the oracle produces results

The proportion of actual faults that the oracle correctly identifies

The total number of test cases executed

Certified Tester AI Testing

Domain 8Objective 7

Test Oracles for AI-Based Systems CT-AI Practice Questions (Page 1)

Part of the Domain 8: Testing AI-Specific Quality Characteristics domain, which makes up ~13% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~3–5 in this domain), expect 1–1 from this objective — we provide 29 practice questions to prepare you well beyond it. (estimate)

29questions here

6free pages

10concepts

Questions 1–5

1application · medium
A financial firm is testing an AI system that predicts daily stock price movements. The system is non-deterministic: running the same input twice can produce slightly different outputs due to stochastic elements. The test team needs an oracle to judge whether a prediction is acceptable. Which oracle approach is most suitable?
Select an answer first
2foundation · easy
Which type of oracle is most suitable for validating AI outputs in a regression task where the distribution of errors should be approximately normal?
Select an answer first
3expert · hard
A team is testing an AI system that predicts whether a loan applicant will default. The system outputs a probability of default. The team has historical data on actual defaults for a test set. They are evaluating two candidate oracles: Oracle A flags applicants with predicted probability > 0.5, and Oracle B flags applicants with predicted probability > 0.7. The team wants to minimize the risk of missing actual defaults (high recall) while keeping the number of false alarms manageable. Which oracle should they choose?
Select an answer first
4foundation · easy
For an AI system that predicts house prices, which oracle type is most appropriate when a large dataset of historical sales is available and the goal is to check that predictions are statistically consistent with past data?
Select an answer first
5foundation · easy
What does the precision of a test oracle measure?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-AI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-AI” is a trademark of its owner, used for identification only.

Test Oracles for AI-Based Systems CT-AI Practice Questions (Page 1)

Questions 1–5

Which type of oracle is most suitable for validating AI outputs in a regression task where the distribution of errors should be approximately normal?

For an AI system that predicts house prices, which oracle type is most appropriate when a large dataset of historical sales is available and the goal is to check that predictions are statistically consistent with past data?

What does the precision of a test oracle measure?

Keep studying — Domain 8: Testing AI-Specific Quality Characteristics