Evaluation Metrics and Model Selection GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   A team is deploying an LLM for a document summarization service. Which metric is most appropriate to monitor the quality of the generated summaries?

   • APerplexity
   • BThroughput
   • CLatency
   • DROUGE
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A company deploys an LLM for a customer support email auto-responder. The system must generate responses that are helpful and resolve the customer's issue. The team is considering using a large model with high accuracy but high latency, or a smaller model with lower latency but slightly lower accuracy. The company has a service-level agreement (SLA) that requires responses within 2 seconds. Which model should the team choose?

   • AThe small model, but only for simple queries and the large model for complex ones
   • BThe small model, because it can meet the latency requirement
   • CThe large model, because accuracy is more important than latency
   • DThe large model, but with a timeout that falls back to the small model
   Reveal Solution Discussion Select an answer first
4. 13expert · hard

   A team is evaluating two LLMs for a machine translation task. Model A has a BLEU score of 30, and Model B has a BLEU score of 35. However, human evaluators rate Model A's translations as more fluent and natural. What is the most likely explanation?

   • ABLEU is not a reliable metric for translation quality
   • BModel A is better because human evaluation is subjective
   • CModel B is better because BLEU is the gold standard
   • DThe BLEU scores are incorrect
   Reveal Solution Discussion Select an answer first
5. 14expert · hard

   A company is deploying an LLM for a legal document review tool. The tool must extract key clauses and summarize them. The team has two models: a 13B-parameter model that runs on a single GPU and a 70B-parameter model that requires multiple GPUs. The 70B model has higher accuracy on a legal benchmark, but the company has a limited budget for GPU infrastructure. The team also needs to process a high volume of documents daily. What is the most cost-effective approach that maintains acceptable accuracy?

   • AUse the 13B model for initial screening and the 70B model for complex cases
   • BUse the 70B model but only for a subset of documents
   • CDeploy the 13B model on a single GPU, accepting lower accuracy
   • DDeploy the 70B model on multiple GPUs to maximize accuracy
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   A team is deploying an LLM-based customer support chatbot. Which metric is most important to monitor to ensure users receive responses quickly?

   • ABLEU
   • BLatency
   • CROUGE
   • DPerplexity
   Reveal Solution Discussion Select an answer first