A team is integrating an LLM-based test oracle into their CI/CD pipeline. They need to ensure that if the model produces an error, the pipeline fails and the team is notified. They also need to be able to debug the issue by reviewing the exact input and output. What should they implement?

Use a model registry to version the model and rely on the registry to handle errors.

Run the model in a separate service and only log the output.

Wrap the model call in a try-catch block and log the error, but continue the pipeline.

Use a CI/CD plugin that runs the model and fails the build on error, while logging the input and output for debugging.

Why is model versioning important in LLM deployment for software testing?

To ensure reproducibility and traceability of test results

To automatically fix bugs in the test code

To increase the speed of test execution

To reduce the cost of LLM inference

A team is developing an LLM-based test oracle. They have collected a dataset of past test results and are now fine-tuning the model. After fine-tuning, they need to ensure the model performs well on new, unseen test cases. What is the next step in the LLM lifecycle?

Version the model and register it in the model registry.

Evaluate the model on a held-out validation set.

Deploy the model directly to production.

Monitor the model in production for drift.

What is the purpose of establishing governance policies for LLM use in software testing?

To reduce the cost of LLM deployment

To ensure LLM usage aligns with organizational and ethical standards

To eliminate the need for human testers

To maximize the number of test cases generated

Certified Tester Testing with Generative AI

Domain 4Objective 5

LLMOps When Deploying and Managing LLMs for Software Testing CT-GENAI Practice Questions (Page 7)

Part of the LLM-Powered Test Infrastructure for Software Testing domain, which makes up ~18% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~5–7 in this domain), expect 1–1 from this objective — we provide 36 practice questions to prepare you well beyond it. (estimate)

36questions here

8free pages

11concepts

11 Concepts in This Objective

LLMOps fundamentals
LLM lifecycle management
Model versioning and registry
Deployment strategies for LLMs
Scaling and resource management
Monitoring and observability
Evaluation and quality assurance
Security and compliance
Cost optimization
Integration with CI/CD
Governance and ethics

Questions 31–35

31expert · medium
A team is integrating an LLM-based test oracle into their CI/CD pipeline. They need to ensure that if the model produces an error, the pipeline fails and the team is notified. They also need to be able to debug the issue by reviewing the exact input and output. What should they implement?
Select an answer first
32foundation · easy
Which deployment option for an LLM is most suitable when test data must not leave the organization's network due to strict data privacy policies?
Select an answer first
33foundation · easy
Why is model versioning important in LLM deployment for software testing?
Select an answer first
34application · medium
A team is developing an LLM-based test oracle. They have collected a dataset of past test results and are now fine-tuning the model. After fine-tuning, they need to ensure the model performs well on new, unseen test cases. What is the next step in the LLM lifecycle?
Select an answer first
35foundation · easy
What is the purpose of establishing governance policies for LLM use in software testing?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-GENAI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-GENAI” is a trademark of its owner, used for identification only.