Use Tools and Metrics to Evaluate Retrieval Performance GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   In the context of retrieval evaluation, what is 'ground truth'?

   • AThe set of documents returned by the retrieval system for a given query
   • BThe query embeddings used to search the document collection
   • CThe manually labeled set of relevant documents for a set of queries
   • DThe raw text corpus from which documents are retrieved
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A team is using the `ragas` library to evaluate retrieval performance in a Databricks notebook. They have a dataset with queries, retrieved documents, and ground-truth relevance labels. They want to compute the `context_precision` and `context_recall` metrics. What is the correct way to use the library?

   • ACall `score()` on each metric individually with the dataset
   • BManually implement the metrics using pandas and numpy
   • CCall `evaluate()` with the dataset and a list of metric objects
   • DUse `compute_metrics()` from the `datasets` library
   Reveal Solution Discussion Select an answer first
4. 13expert · medium

   A team is evaluating a RAG pipeline for a financial compliance system. They have a ground-truth set of 200 queries with relevance judgments. The retriever currently returns 5 documents per query. They compute recall@5 = 0.9 and precision@5 = 0.6. The compliance team requires that the final answer must be based on the most relevant document, and they are concerned about the impact of irrelevant documents in the context. They want to improve the pipeline's ability to surface the single most relevant document at the top of the list. Which metric should they focus on improving?

   • ARecall@5
   • BNormalized Discounted Cumulative Gain (NDCG)
   • CMean Reciprocal Rank (MRR)
   • DPrecision@5
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   A RAG pipeline retrieves 10 documents for each query. A human evaluator labels each retrieved document as relevant or not. For a particular query, 4 of the 10 retrieved documents are relevant, and the system's ranked list places 2 relevant documents in the top 3 positions. The team wants a single metric that captures both how many relevant documents were retrieved and how highly they were ranked. Which metric should they use?

   • ARecall@10
   • BNormalized Discounted Cumulative Gain (NDCG)
   • CMean Reciprocal Rank (MRR)
   • DPrecision@3
   Reveal Solution Discussion Select an answer first
6. 15application · medium

   A RAG pipeline for a customer support chatbot retrieves 5 documents per query. The team evaluates the retriever on a test set and finds that precision@5 is 0.6, but recall@5 is 0.4. They want to improve the retriever so that it retrieves more of the relevant documents, even if it means retrieving some irrelevant ones. Which change is most aligned with this goal?

   • ASwitch to a retriever that only returns the single most relevant document
   • BDecrease the number of documents retrieved per query from 5 to 3
   • CIncrease the number of documents retrieved per query from 5 to 10
   • DTighten the similarity threshold to only retrieve highly similar documents
   Reveal Solution Discussion Select an answer first