A test team uses an LLM to generate test cases for a loan approval system. The team is concerned about bias against certain ethnic groups. They run fairness metrics and find that the test cases are balanced across ethnic groups. However, they also notice that the LLM generates test cases that assume all applicants have a stable employment history, which may not be representative. The team needs to decide whether to use the test cases. Which consideration is most important?

The test cases should be used because they reflect the majority of loan applicants

The test cases are balanced, so they are unbiased and ready to use

The fairness metrics only measure demographic parity, not other forms of bias such as socioeconomic bias

The test cases should be discarded because they are generated by an LLM

Which method is effective for detecting reasoning errors in LLM outputs?

Checking the output for spelling mistakes

Validating each step of the model's reasoning process

Ensuring the output is short and concise

Verifying the output against a dictionary

What is confirmation bias in the context of LLM outputs?

The model only uses confirmatory evidence from its training data.

The model is biased against confirming any information.

The model always confirms that the user's input is correct.

The model tends to favor information that confirms a user's existing beliefs.

A test team uses an LLM to generate test data for a banking application. The LLM produces a list of valid-looking but non-existent SWIFT/BIC codes for international transfers. The team needs to determine whether these are hallucinations or correct responses. Which action is most appropriate?

Ask the LLM to regenerate the list multiple times and compare outputs for consistency

Use the codes in test data but flag them as 'synthetic' to avoid confusion

Accept the codes as correct because the LLM is trained on vast financial data

Cross-check the generated codes against an authoritative SWIFT/BIC registry

Which technique is commonly used to detect hallucinations in LLM outputs?

Measuring the length of the output

Checking the output against a reliable external source

Ensuring the output is grammatically correct

Checking if the output contains any numbers

Certified Tester Testing with Generative AI

Domain 3Objective 1

Identify Hallucinations, Reasoning Errors and Biases in LLM Output CT-GENAI Practice Questions (Page 5)

Part of the Managing Risks of Generative AI in Software Testing domain, which makes up ~18% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~5–7 in this domain), expect 1–1 from this objective — we provide 37 practice questions to prepare you well beyond it. (estimate)

37questions here

8free pages

12concepts

Questions 21–25

21expert · hard
A test team uses an LLM to generate test cases for a loan approval system. The team is concerned about bias against certain ethnic groups. They run fairness metrics and find that the test cases are balanced across ethnic groups. However, they also notice that the LLM generates test cases that assume all applicants have a stable employment history, which may not be representative. The team needs to decide whether to use the test cases. Which consideration is most important?
Select an answer first
22foundation · easy
Which method is effective for detecting reasoning errors in LLM outputs?
Select an answer first
23foundation · easy
What is confirmation bias in the context of LLM outputs?
Select an answer first
24application · medium
A test team uses an LLM to generate test data for a banking application. The LLM produces a list of valid-looking but non-existent SWIFT/BIC codes for international transfers. The team needs to determine whether these are hallucinations or correct responses. Which action is most appropriate?
Select an answer first
25foundation · easy
Which technique is commonly used to detect hallucinations in LLM outputs?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-GENAI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-GENAI” is a trademark of its owner, used for identification only.