A team is evaluating AI-generated outputs for two different test tasks: generating test cases and writing defect reports. They want to use appropriate metrics for each task. Which combination of metrics is most suitable?

Consistency for test cases; relevance for defect reports

Coverage for test cases; factual accuracy for defect reports

Readability for test cases; coverage for defect reports

A team uses generative AI to generate test data for a security testing scenario. The relevance score is high, but the generation process is slow and expensive. The team has a limited budget and needs the data for a one-time security audit. What should they do?

Generate the full dataset despite the cost and time

Generate a smaller dataset that covers the most critical security scenarios

Skip the generative AI and manually create the test data

Use a cheaper but less relevant generative AI service

What is a key outcome of analyzing metric results for generated test data?

Identifying whether the test data covers the necessary edge cases

Reducing the cost of the AI model

Guaranteeing that the test data is free from any errors

Automatically generating more test data

A team uses generative AI to generate test cases for a complex workflow. The consistency metric shows high variability across runs, but the factual accuracy of each run is high. The team needs to use the test cases for regression testing, which requires stable and repeatable test suites. What should they do?

Run the generation multiple times and merge all unique test cases

Use a fixed random seed and a more specific prompt to reduce variability

Accept the variability because accuracy is high

Manually review and select the best test cases from each run

A team uses generative AI to generate test cases for a medical device software. The automated metrics show high coverage and accuracy, but the team is concerned about the usability of the test cases for the domain experts who will execute them. What should they do to evaluate the test cases?

Use the test cases without review to save time

Rely solely on the automated metrics because they are high

Have domain experts review the test cases for clarity, completeness, and alignment with clinical workflows

Compare the test cases to those from a previous project

Certified Tester Testing with Generative AI

Domain 2Objective 8

Metrics for Evaluating the Results of Generative AI on Test Tasks CT-GENAI Practice Questions (Page 2)

Part of the Prompt Engineering for Effective Software Testing domain, which makes up ~30% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~8–12 in this domain), expect 1–1 from this objective — we provide 35 practice questions to prepare you well beyond it. (estimate)

35questions here

7free pages

9concepts

Questions 6–10

6application · medium
A team is evaluating AI-generated outputs for two different test tasks: generating test cases and writing defect reports. They want to use appropriate metrics for each task. Which combination of metrics is most suitable?
Select an answer first
7expert · hard
A team uses generative AI to generate test data for a security testing scenario. The relevance score is high, but the generation process is slow and expensive. The team has a limited budget and needs the data for a one-time security audit. What should they do?
Select an answer first
8foundation · easy
What is a key outcome of analyzing metric results for generated test data?
Select an answer first
9expert · hard
A team uses generative AI to generate test cases for a complex workflow. The consistency metric shows high variability across runs, but the factual accuracy of each run is high. The team needs to use the test cases for regression testing, which requires stable and repeatable test suites. What should they do?
Select an answer first
10expert · hard
A team uses generative AI to generate test cases for a medical device software. The automated metrics show high coverage and accuracy, but the team is concerned about the usability of the test cases for the domain experts who will execute them. What should they do to evaluate the test cases?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-GENAI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-GENAI” is a trademark of its owner, used for identification only.