A company is developing an NLP model to summarize legal documents. They are choosing between two benchmark suites: one is a general summarization benchmark (e.g., CNN/Daily Mail) and another is a legal-specific benchmark (e.g., LexGLUE). They have limited compute resources and need to evaluate quickly. What is the best approach?

Skip benchmarks and evaluate only on a small internal dataset.

Use the legal-specific benchmark because it better represents the target domain, even if it takes longer.

Use both benchmarks and average the results.

Use the general benchmark because it is faster to run and widely recognized.

A model trained on ImageNet achieves 95% top-5 accuracy on the benchmark. However, when deployed in a self-driving car to recognize pedestrians, it fails frequently. What is the most likely reason for this discrepancy?

The model is overfitting to ImageNet and does not generalize to real-world driving scenes.

ImageNet is too easy, so high accuracy is expected.

ImageNet does not include any images of pedestrians.

The top-5 metric is not appropriate for this task.

What is the primary purpose of using benchmark suites when evaluating machine learning models?

To replace the need for domain-specific testing and validation

To guarantee that a model will perform perfectly in all real-world conditions

To automatically optimize the hyperparameters of any machine learning model

To provide a standardized way to compare the performance of different models on the same task

A company is building a model to detect fraudulent credit card transactions. They have a highly imbalanced dataset. Which benchmark suite is most appropriate to evaluate their model?

A tabular benchmark like the UCI Credit Card dataset

Certified Tester AI Testing

Domain 5Objective 4

Benchmark Suites for ML CT-AI Practice Questions (Page 4)

Part of the Domain 5: ML Functional Performance Metrics domain, which makes up ~5% of our current practice bank. ISTQB does not publish an official question count, but from its 60-minute exam (~25–40 total, ~1–2 in this domain), expect 1–1 from this objective — we provide 23 practice questions to prepare you well beyond it. (estimate)

23questions here

5free pages

5concepts

Questions 16–20

16expert · hard
A company is developing an NLP model to summarize legal documents. They are choosing between two benchmark suites: one is a general summarization benchmark (e.g., CNN/Daily Mail) and another is a legal-specific benchmark (e.g., LexGLUE). They have limited compute resources and need to evaluate quickly. What is the best approach?
Select an answer first
17expert · hard
A model trained on ImageNet achieves 95% top-5 accuracy on the benchmark. However, when deployed in a self-driving car to recognize pedestrians, it fails frequently. What is the most likely reason for this discrepancy?
Select an answer first
18foundation · easy
What is the primary purpose of using benchmark suites when evaluating machine learning models?
Select an answer first
19application · medium
A company is building a model to detect fraudulent credit card transactions. They have a highly imbalanced dataset. Which benchmark suite is most appropriate to evaluate their model?
Select an answer first
20application · easy
A team is working on a question-answering system and needs a benchmark to evaluate their model. Which benchmark is most appropriate?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CT-AI

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CT-AI” is a trademark of its owner, used for identification only.

Benchmark Suites for ML CT-AI Practice Questions (Page 4)

Questions 16–20

A model trained on ImageNet achieves 95% top-5 accuracy on the benchmark. However, when deployed in a self-driving car to recognize pedestrians, it fails frequently. What is the most likely reason for this discrepancy?

What is the primary purpose of using benchmark suites when evaluating machine learning models?

A company is building a model to detect fraudulent credit card transactions. They have a highly imbalanced dataset. Which benchmark suite is most appropriate to evaluate their model?

A team is working on a question-answering system and needs a benchmark to evaluate their model. Which benchmark is most appropriate?

Keep studying — Domain 5: ML Functional Performance Metrics