Use Tools and Metrics to Evaluate Retrieval Performance GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A team is creating a ground-truth dataset for evaluating a RAG system that retrieves legal documents. They have a set of queries and a large pool of candidate documents. To ensure the evaluation is reliable, they want to minimize the risk of missing relevant documents that the retriever might not return. What is the best practice for constructing the relevance judgments?

   • AHave a single annotator label all documents for all queries to ensure consistency
   • BAutomatically label documents as relevant if they share high word overlap with the query
   • CUse a pooling method where multiple retrieval systems contribute candidate documents for human labeling
   • DLabel only the documents that the current retriever returns for each query
   Reveal Solution Discussion Select an answer first
2. 2expert · medium

   A team is evaluating a RAG pipeline using the `ragas` library. They have a dataset with 100 queries, each with a list of retrieved documents and ground-truth relevance labels. They run the evaluation and get a `context_precision` of 0.8 and a `context_recall` of 0.5. They want to improve the retriever to increase recall without significantly hurting precision. Which change is most likely to achieve this?

   • AIncrease the number of retrieved documents per query
   • BTighten the similarity threshold to only retrieve highly similar documents
   • CSwitch to a retriever that only returns the single most relevant document
   • DDecrease the number of retrieved documents per query
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A team is building a retrieval evaluation set for a RAG system that answers customer support questions. They have a set of queries and a pool of candidate documents. To compute retrieval metrics, they need to label which documents are relevant to each query. What is the most appropriate way to create these labels?

   • AUse the documents that the language model cites in its generated answers as the relevant set
   • BUse the documents that the current retriever returns for each query as the relevant set
   • CAutomatically label all documents that contain the query's keywords as relevant
   • DHave domain experts manually judge each document in the pool for relevance to each query
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   When using a tool like pytrec_eval to compute retrieval metrics, what input data is typically required?

   • AThe embedding vectors of the queries and documents
   • BThe relevance judgments (ground truth) and the ranked lists of retrieved documents for each query
   • CThe prompts and generated answers from the LLM
   • DThe raw text of all documents in the corpus
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   What is a 'relevance judgment' in retrieval evaluation?

   • AThe order in which documents are returned by the retrieval system
   • BA binary or graded label indicating whether a document is relevant to a query
   • CThe similarity score between a query and a document embedding
   • DA score assigned by the retrieval model indicating how confident it is in a result
   Reveal Solution Discussion Select an answer first