Benchmark Suites for ML CT-AI Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   A model achieves state-of-the-art results on the SuperGLUE benchmark. However, when deployed in a customer service chatbot, it performs poorly on real user queries. What is the most likely cause?

   • AThe benchmark tasks do not represent the informal, varied language of real customer queries.
   • BThe model is not large enough to handle real-world data.
   • CSuperGLUE is too difficult, so the model is overfitting.
   • DSuperGLUE does not provide a standard metric, so the results are unreliable.
   Reveal Solution Discussion Select an answer first
2. 12application · easy

   A team is developing a new image classification model and wants to compare it with existing models in a standardized way. They decide to use a benchmark suite. What is the primary purpose of using a benchmark suite in this context?

   • ATo train the model on a large dataset
   • BTo provide a common set of data and metrics for fair comparison
   • CTo guarantee the model will work in production
   • DTo automatically tune hyperparameters
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A team is presenting their model's performance to stakeholders. They claim their model is better than others because it has a higher score on a benchmark. What is a key caveat they should mention?

   • AThe benchmark score is the only measure of model quality.
   • BThe benchmark score is always reproducible.
   • CThe benchmark score is independent of the training data.
   • DThe benchmark may not reflect the model's performance in the real-world deployment context.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Which of the following is a known limitation of benchmark datasets when representing real-world scenarios?

   • AThey are designed to be biased towards a specific model architecture
   • BThey are typically too large to be used in practice
   • CThey are always perfectly labeled and free of errors
   • DThey may not capture the full diversity and variability of real-world data
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A model performs well on the CIFAR-10 benchmark but poorly on a custom dataset of satellite images. What is the most likely reason?

   • ACIFAR-10 uses a different metric than the custom dataset.
   • BThe satellite images have a different distribution than CIFAR-10 images.
   • CCIFAR-10 is too small, so the model is underfitting.
   • DThe model is overfitting to CIFAR-10.
   Reveal Solution Discussion Select an answer first