Evaluating RAG Beyond Plausible Answers

Decompose RAG failures across retrieval, context construction, and generation so each result points to an actionable change.

阅读中文版 →
On this page

The most misleading property of a RAG system is that a fluent answer can look correct even when the supporting evidence was never retrieved. Fluency is not faithfulness, and neither tells us where the system failed.

A useful evaluation separates the pipeline into three questions: did retrieval find the evidence, did context construction preserve it, and did generation use it correctly?

Separate the failure modes

I usually start with three layers:

  1. Retrieval: Did a relevant source enter the candidate set?
  2. Context: Did the useful passage survive ranking, truncation, and formatting?
  3. Generation: Is the answer grounded, complete, and willing to abstain when evidence is missing?

Retrieval metrics such as Recall@K and MRR cannot replace answer-level evaluation. Likewise, a strong faithfulness score cannot rescue a retriever that consistently misses the right document.

A minimal retrieval check

from collections.abc import Iterable


def hit_at_k(retrieved: Iterable[str], relevant: set[str], k: int) -> int:
    top_k = list(retrieved)[:k]
    return int(any(doc_id in relevant for doc_id in top_k))


def recall_at_k(retrieved: Iterable[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return 0.0
    top_k = set(list(retrieved)[:k])
    return len(top_k & relevant) / len(relevant)

The formula is easy. The hard part is labeling relevance. Some questions require two passages together, while others have several equally valid sources. A benchmark with one “correct document” can punish a perfectly reasonable retrieval result.

Keep slices, not only averages

An average score hides the structure of failure. I want to see results by question type, source freshness, query wording, access level, and answerability. When paraphrased queries fail, query understanding may be the bottleneck. When multi-hop questions fail, retrieval strategy or context assembly may matter more than model size.

Treat evaluation as a development interface

Evaluation should run whenever retrieval, chunking, prompts, or models change. The useful questions are not only “What is the score?” but also: which requests improved, what regressed, and do we understand why?

When those questions can be answered consistently, a RAG demo begins to become a maintainable system.

RELATED WRITING

DISCUSSION

Discussion