Evaluation Metrics and Model Selection GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A team is evaluating two LLMs for a summarization task. Model A has a higher ROUGE-L score but a lower BERTScore, while Model B has a lower ROUGE-L but a higher BERTScore. The summaries are for a medical research digest where semantic accuracy is critical. Which model should the team choose?

   • AModel A, because higher ROUGE-L indicates better factual accuracy
   • BModel B, because lower ROUGE-L is always better
   • CModel B, because BERTScore better captures semantic similarity
   • DModel A, because ROUGE-L is the standard metric for summarization
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A healthcare organization deploys an LLM to summarize patient discharge notes. The summaries are reviewed by clinicians before being added to the electronic health record. The organization must ensure that the summaries do not omit critical information such as medication changes or follow-up instructions. Which metric should be the primary focus during evaluation?

   • AThroughput
   • BROUGE-L
   • CRecall of key clinical entities
   • DPerplexity
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A research team is evaluating two LLMs for a text summarization task. They have a reference summary for each document. The team wants a metric that measures how much of the key information from the reference is captured in the generated summary, while also being robust to synonyms and paraphrasing. Which metric is best suited?

   • AROUGE-L
   • BBERTScore
   • CBLEU
   • DPerplexity
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   Which metric is specifically designed to evaluate the quality of machine-translated text by comparing it to one or more reference translations?

   • ALatency
   • BPerplexity
   • CROUGE
   • DBLEU
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   A team needs to deploy an LLM for a real-time chat application with strict latency requirements and limited GPU memory. Which model size and architecture choice best balances performance and resource constraints?

   • AA large dense model with 70B parameters
   • BA smaller dense model with 7B parameters
   • CA mixture-of-experts model with 70B total parameters
   • DA model with 175B parameters and quantization
   Reveal Solution Discussion Select an answer first