What is a common pitfall when fine-tuning an LLM on a small dataset of test cases?

The model may overfit and memorize the training examples, performing poorly on new inputs

The model will always produce invalid test cases

The model will become too large to deploy

The model will require more training data than is available

Which activity is part of the evaluation step in the fine-tuning process?

Adjusting the model's architecture

Testing the fine-tuned model on a held-out validation set

Deploying the model to a staging environment

A team is fine-tuning an LLM to generate test cases for a large legacy system. They have a dataset of 10,000 test cases, but the dataset is imbalanced: 90% are for the login module, and 10% cover the rest of the system. They want the model to generate test cases for all modules. What is the most effective strategy?

Train on the imbalanced dataset as-is

Evaluate the model only on the login module

Use a higher learning rate to force the model to learn minority classes

Balance the dataset by undersampling the majority class and oversampling the minority classes

Certified Tester Testing with Generative AI

Domain 4Objective 4

Fine-Tuning LLMs for Test Tasks CT-GENAI Practice Questions (Page 4)

Part of the LLM-Powered Test Infrastructure for Software Testing domain, which makes up ~18% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~5–7 in this domain), expect 1–1 from this objective — we provide 28 practice questions to prepare you well beyond it. (estimate)

28questions here

6free pages

6concepts

Questions 16–20

16foundation · easy
What is a common pitfall when fine-tuning an LLM on a small dataset of test cases?
Select an answer first
17foundation · easy
Which activity is part of the evaluation step in the fine-tuning process?
Select an answer first
18expert · hard
A team is fine-tuning an LLM to generate test cases for a large legacy system. They have a dataset of 10,000 test cases, but the dataset is imbalanced: 90% are for the login module, and 10% cover the rest of the system. They want the model to generate test cases for all modules. What is the most effective strategy?
Select an answer first
19expert · hard
A team has fine-tuned an LLM to generate test cases for a web application. They have a test set of 200 test cases. The model generated 250 test cases, of which 150 are correct and relevant, 50 are correct but not relevant, and 50 are incorrect. What is the recall of the model on this test set?
Select an answer first
20expert · hard
A team has fine-tuned an LLM to generate test cases for a web application. They have a test set of 100 test cases. The model generated 120 test cases, of which 80 are correct and relevant, 20 are correct but not relevant, and 20 are incorrect. What is the precision of the model on this test set?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-GENAI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-GENAI” is a trademark of its owner, used for identification only.

Fine-Tuning LLMs for Test Tasks CT-GENAI Practice Questions (Page 4)

Questions 16–20

What is a common pitfall when fine-tuning an LLM on a small dataset of test cases?

Which activity is part of the evaluation step in the fine-tuning process?

A team has fine-tuned an LLM to generate test cases for a web application. They have a test set of 200 test cases. The model generated 250 test cases, of which 150 are correct and relevant, 50 are correct but not relevant, and 50 are incorrect. What is the recall of the model on this test set?

A team has fine-tuned an LLM to generate test cases for a web application. They have a test set of 100 test cases. The model generated 120 test cases, of which 80 are correct and relevant, 20 are correct but not relevant, and 20 are incorrect. What is the precision of the model on this test set?

Keep studying — LLM-Powered Test Infrastructure for Software Testing