A team runs the same generative AI prompt three times to generate test cases for a login feature. They notice that each run produces a different set of test cases, some missing critical negative tests. Which metric should they use to quantify this variability?

Requirement coverage percentage

Semantic similarity to the prompt

Output consistency across repeated runs

After evaluating generated test cases with multiple metrics, what is the primary purpose of interpreting the metric results?

To increase the speed of the AI model

To replace the need for human testers

To automatically fix any defects in the generated test cases

To determine whether the outputs are suitable for use in testing and to guide improvements

A team uses generative AI to produce defect reports from crash logs. They need to verify that the reported root cause and affected component match the actual code behavior. Which metric is most relevant for this evaluation?

Readability score of the report

Time taken to generate the report

Factual accuracy of the defect description

Number of similar defects found in the past

What does a reliability metric for generative AI outputs primarily measure?

The speed of generating outputs

The number of test cases generated

The stability of outputs across different runs

The relevance of outputs to the test task

Certified Tester Testing with Generative AI

Domain 2Objective 8

Metrics for Evaluating the Results of Generative AI on Test Tasks CT-GENAI Practice Questions (Page 4)

Part of the Prompt Engineering for Effective Software Testing domain, which makes up ~30% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~8–12 in this domain), expect 1–1 from this objective — we provide 35 practice questions to prepare you well beyond it. (estimate)

35questions here

7free pages

9concepts

Questions 16–20

16application · medium
A team runs the same generative AI prompt three times to generate test cases for a login feature. They notice that each run produces a different set of test cases, some missing critical negative tests. Which metric should they use to quantify this variability?
Select an answer first
17foundation · easy
After evaluating generated test cases with multiple metrics, what is the primary purpose of interpreting the metric results?
Select an answer first
18application · medium
A team uses generative AI to produce defect reports from crash logs. They need to verify that the reported root cause and affected component match the actual code behavior. Which metric is most relevant for this evaluation?
Select an answer first
19foundation · easy
Which metric evaluates whether a generated test case is directly applicable to the specific feature being tested?
Select an answer first
20foundation · easy
What does a reliability metric for generative AI outputs primarily measure?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-GENAI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-GENAI” is a trademark of its owner, used for identification only.