A team is evaluating two different oracles for an AI system that classifies customer reviews as positive or negative. Oracle X is a human expert who labels a sample of reviews. Oracle Y is a heuristic that flags reviews containing certain negative keywords. The team wants to assess the reliability of each oracle. Which approach is most appropriate for evaluating oracle quality?

Use a third oracle, such as a statistical model, to arbitrate disagreements between Oracle X and Oracle Y.

Compare the outputs of Oracle X and Oracle Y on the same set of reviews and measure their agreement.

Rely on the oracle that is faster and cheaper, assuming it is sufficient.

Evaluate the precision and recall of each oracle against a ground truth dataset of manually labeled reviews.

A team is testing a spam-filtering AI that classifies emails as spam or not spam. They have a large unlabeled dataset and limited budget for human review. They want to maximize the chance of catching misclassifications. Which combination of oracles is most effective?

Use a model-based oracle that compares the spam filter's output to a previously trained, frozen version of the same model.

Use a human oracle to review every email in the dataset.

Use only a heuristic oracle that flags emails containing typical spam keywords.

Use a statistical oracle to check the overall spam-to-ham ratio and a heuristic oracle to flag emails with high confidence but unusual content.

A team is evaluating the quality of a test oracle for an AI system that detects fraudulent transactions. The oracle flags transactions as 'suspicious' if the AI's fraud score exceeds a threshold. The team wants to assess how well this oracle detects actual fraud. Which metric is most appropriate for evaluating the oracle's ability to catch true fraud cases?

F1-score, which is the harmonic mean of precision and recall.

Recall, which measures the proportion of actual fraudulent transactions that are flagged by the oracle.

Precision, which measures the proportion of flagged transactions that are actually fraudulent.

Accuracy, which measures the overall proportion of correct classifications.

A team is testing an AI system that classifies satellite images into land-use categories (e.g., urban, forest, water). They have a labeled dataset for training but want to test on new, unlabeled images. They also have a legacy classification model that is known to be reasonably accurate but not perfect. Which combination of oracles would provide the most robust validation?

Use a statistical oracle to check that the distribution of predicted classes matches the known distribution of land-use types.

Use only the legacy model as an oracle, comparing its classifications to the new AI's classifications.

Use a heuristic oracle that flags images where the AI's confidence is low.

Use the legacy model as a model-based oracle, and use a statistical oracle to check the distribution of disagreements.

Certified Tester AI Testing

Domain 8Objective 7

Test Oracles for AI-Based Systems CT-AI Practice Questions (Page 5)

Part of the Domain 8: Testing AI-Specific Quality Characteristics domain, which makes up ~13% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~3–5 in this domain), expect 1–1 from this objective — we provide 29 practice questions to prepare you well beyond it. (estimate)

29questions here

6free pages

10concepts

Questions 21–25

21expert · hard
A team is evaluating two different oracles for an AI system that classifies customer reviews as positive or negative. Oracle X is a human expert who labels a sample of reviews. Oracle Y is a heuristic that flags reviews containing certain negative keywords. The team wants to assess the reliability of each oracle. Which approach is most appropriate for evaluating oracle quality?
Select an answer first
22foundation · easy
Which type of test oracle relies on the judgment of human experts to determine whether an AI system's output is correct?
Select an answer first
23application · medium
A team is testing a spam-filtering AI that classifies emails as spam or not spam. They have a large unlabeled dataset and limited budget for human review. They want to maximize the chance of catching misclassifications. Which combination of oracles is most effective?
Select an answer first
24application · medium
A team is evaluating the quality of a test oracle for an AI system that detects fraudulent transactions. The oracle flags transactions as 'suspicious' if the AI's fraud score exceeds a threshold. The team wants to assess how well this oracle detects actual fraud. Which metric is most appropriate for evaluating the oracle's ability to catch true fraud cases?
Select an answer first
25application · medium
A team is testing an AI system that classifies satellite images into land-use categories (e.g., urban, forest, water). They have a labeled dataset for training but want to test on new, unlabeled images. They also have a legacy classification model that is known to be reasonably accurate but not perfect. Which combination of oracles would provide the most robust validation?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-AI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-AI” is a trademark of its owner, used for identification only.

Test Oracles for AI-Based Systems CT-AI Practice Questions (Page 5)

Questions 21–25

Which type of test oracle relies on the judgment of human experts to determine whether an AI system's output is correct?

A team is testing a spam-filtering AI that classifies emails as spam or not spam. They have a large unlabeled dataset and limited budget for human review. They want to maximize the chance of catching misclassifications. Which combination of oracles is most effective?

Keep studying — Domain 8: Testing AI-Specific Quality Characteristics