A team uses generative AI to create test cases for a user interface. They want to ensure the test cases are easy to understand and maintain by the test automation engineers. Which evaluation approach is most appropriate?

Compare test cases to those generated by a different tool

Have test automation engineers rate the clarity and maintainability of each test case

Check that all test cases use the same locator strategy

Measure the number of test cases generated

What does a performance metric for a generative AI model in testing primarily evaluate?

The correctness of the generated test cases

The coverage of requirements by the test cases

The response time and resource usage of the model

The consistency of outputs across runs

What is the primary purpose of evaluation metrics in the context of generative AI applied to software testing tasks?

To provide a quantitative basis for assessing the quality and suitability of generated outputs for testing purposes

To guarantee that generated test cases are free from any defects or errors

To replace human testers by automatically determining the correctness of all generated test artifacts

To measure the performance of the underlying hardware running the AI model

After evaluating a set of AI-generated test cases, a team finds that the factual accuracy is high (95%) but the relevance score is low (40%) because many test cases target features not in the current sprint. What is the most appropriate decision based on these metrics?

Use the test cases but only for regression testing

Accept the test cases because factual accuracy is high

Increase the number of test cases generated

Reject the test cases and refine the prompt to focus on the sprint scope

Certified Tester Testing with Generative AI

Domain 2Objective 8

Metrics for Evaluating the Results of Generative AI on Test Tasks CT-GENAI Practice Questions (Page 5)

Part of the Prompt Engineering for Effective Software Testing domain, which makes up ~30% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~8–12 in this domain), expect 1–1 from this objective — we provide 35 practice questions to prepare you well beyond it. (estimate)

35questions here

7free pages

9concepts

Questions 21–25

21application · medium
A team uses generative AI to create test cases for a user interface. They want to ensure the test cases are easy to understand and maintain by the test automation engineers. Which evaluation approach is most appropriate?
Select an answer first
22foundation · easy
What does a performance metric for a generative AI model in testing primarily evaluate?
Select an answer first
23foundation · easy
Which metric measures the computational cost of generating test cases using a generative AI model?
Select an answer first
24foundation · easy
What is the primary purpose of evaluation metrics in the context of generative AI applied to software testing tasks?
Select an answer first
25application · medium
After evaluating a set of AI-generated test cases, a team finds that the factual accuracy is high (95%) but the relevance score is low (40%) because many test cases target features not in the current sprint. What is the most appropriate decision based on these metrics?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-GENAI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-GENAI” is a trademark of its owner, used for identification only.