RAG Evaluation

Why Evaluate RAG?

A RAG system can produce a fluent answer and still be wrong. The failure may happen because the right chunk was never retrieved, the retrieved context was noisy, or the model used the evidence incorrectly.

The Two Main Evaluation Layers

A strong generator cannot reliably fix missing evidence. If the retriever misses the required policy clause, the model may have nothing trustworthy to use.

Build an Evaluation Dataset

Start with a representative set of questions. Each test case should describe what evidence is expected and, when practical, what a correct answer should contain.

Recall@K

Recall@K asks whether the relevant evidence appears anywhere in the top K retrieved results.

For a binary relevance test, a simple form is: Recall@K = successful queries with relevant evidence in top K ÷ total queries.

Precision@K

Precision@K measures how much of the top-K result set is relevant. It is useful when irrelevant chunks create noise or consume context.

Recall and precision answer different questions: recall asks “did we find it?” while precision asks “how much of what we returned is useful?”

Mean Reciprocal Rank (MRR)

MRR rewards systems that place the first relevant result near the top. For a query, reciprocal rank is 1 ÷ rank of the first relevant result.

MRR is especially useful when the user needs one strong piece of evidence and the ordering of the first relevant result matters.

NDCG: Ranking Quality

Normalized Discounted Cumulative Gain (NDCG) is useful when several retrieved results have different degrees of relevance. Highly relevant results should appear earlier than marginally useful results.

Calculate Retrieval Metrics in Python

You can begin evaluation without a vector database. Represent your retrieved chunk IDs and the known relevant IDs, then calculate metrics over your test set.

def recall_at_k(retrieved, relevant, k):
    top_k = retrieved[:k]
    return int(any(item in relevant for item in top_k))


def precision_at_k(retrieved, relevant, k):
    top_k = retrieved[:k]
    if not top_k:
        return 0.0
    return sum(item in relevant for item in top_k) / len(top_k)


retrieved = ["chunk-8", "chunk-3", "chunk-4", "chunk-12", "chunk-9"]
relevant = {"chunk-4"}

print("Recall@5:", recall_at_k(retrieved, relevant, 5))
print("Precision@5:", precision_at_k(retrieved, relevant, 5))

Evaluate Context Relevance

Even when retrieval technically succeeds, the final context may contain too much irrelevant or repetitive information. Context relevance asks whether the evidence supplied to the model actually helps answer the question.

Faithfulness: Is the Answer Supported?

Faithfulness asks whether the claims in the generated answer are supported by the retrieved context. A fluent answer can still contain unsupported claims.

Answer Correctness

Answer correctness asks whether the final response actually answers the user's question accurately. Depending on the task, you can compare against a reference answer, structured facts, or human judgment.

Correctness and faithfulness are related but not identical. An answer can be plausible and correct by coincidence, or contain the right conclusion plus unsupported details.

Source Attribution

For many RAG applications, users need to know where an answer came from. Evaluate whether citations point to the correct document, page, section, or URL.

Separate Retrieval and Generation Failures

Evaluate Different K Values

Do not assume one Top-K value is optimal. Compare retrieval quality and downstream answer quality at several K values.

Compare Retrieval Strategies

Evaluation should compare system changes using the same test set. This makes improvements measurable instead of anecdotal.

Human Evaluation

Automated metrics are valuable, but humans are still useful for ambiguous questions, nuanced answers, and failure analysis.

Keep the rubric stable when comparing two versions of the system.

LLM-as-a-Judge

An LLM can help score large evaluation sets against a rubric, but its scores should be treated as evaluation signals rather than unquestionable truth.

Track Cost and Latency Too

A RAG change is not automatically an improvement if quality increases slightly while latency and cost increase dramatically.

Build a Repeatable Evaluation Loop

Keep evaluation data versioned. When chunking, embeddings, retrieval, reranking, prompts, or models change, rerun the same benchmark so you can compare the effect.

A Simple Python Evaluation Harness

Here is a small pattern for running the same retrieval test across multiple questions.

def evaluate_retrieval(cases, k=5):
    recall_scores = []
    precision_scores = []

    for case in cases:
        retrieved = case["retrieved"]
        relevant = set(case["relevant"])
        recall_scores.append(recall_at_k(retrieved, relevant, k))
        precision_scores.append(precision_at_k(retrieved, relevant, k))

    return {
        "recall_at_k": sum(recall_scores) / len(recall_scores),
        "precision_at_k": sum(precision_scores) / len(precision_scores),
    }

cases = [
    {"retrieved": ["a", "b", "c"], "relevant": ["c"]},
    {"retrieved": ["x", "y", "z"], "relevant": ["q"]},
]

print(evaluate_retrieval(cases, k=3))

What Good RAG Evaluation Looks Like

Common Evaluation Mistakes

Evaluation Checklist