The most misleading property of a RAG system is that a fluent answer can look correct even when the supporting evidence was never retrieved. Fluency is not faithfulness, and neither tells us where the system failed.
A useful evaluation separates the pipeline into three questions: did retrieval find the evidence, did context construction preserve it, and did generation use it correctly?
Separate the failure modes
I usually start with three layers:
- Retrieval: Did a relevant source enter the candidate set?
- Context: Did the useful passage survive ranking, truncation, and formatting?
- Generation: Is the answer grounded, complete, and willing to abstain when evidence is missing?
Retrieval metrics such as Recall@K and MRR cannot replace answer-level evaluation. Likewise, a strong faithfulness score cannot rescue a retriever that consistently misses the right document.
A minimal retrieval check
from collections.abc import Iterable
def hit_at_k(retrieved: Iterable[str], relevant: set[str], k: int) -> int:
top_k = list(retrieved)[:k]
return int(any(doc_id in relevant for doc_id in top_k))
def recall_at_k(retrieved: Iterable[str], relevant: set[str], k: int) -> float:
if not relevant:
return 0.0
top_k = set(list(retrieved)[:k])
return len(top_k & relevant) / len(relevant)
The formula is easy. The hard part is labeling relevance. Some questions require two passages together, while others have several equally valid sources. A benchmark with one “correct document” can punish a perfectly reasonable retrieval result.
Keep slices, not only averages
An average score hides the structure of failure. I want to see results by question type, source freshness, query wording, access level, and answerability. When paraphrased queries fail, query understanding may be the bottleneck. When multi-hop questions fail, retrieval strategy or context assembly may matter more than model size.
Treat evaluation as a development interface
Evaluation should run whenever retrieval, chunking, prompts, or models change. The useful questions are not only “What is the score?” but also: which requests improved, what regressed, and do we understand why?
When those questions can be answered consistently, a RAG demo begins to become a maintainable system.
DISCUSSION