Improving RAG

Start With the Failure, Not the Tool

“Improve RAG” does not mean adding more components. First identify what is failing. If the correct evidence never enters the candidate set, changing the generation prompt will not repair retrieval.

Improve the Source Data First

RAG quality is bounded by the information you index. Duplicate, stale, incomplete, or badly extracted documents create problems before retrieval even starts.

Improve Chunking

Chunking determines the units that retrieval can return. A chunk that is too small may lose the surrounding meaning; a chunk that is too large may contain unrelated material.

Evaluate chunking with the same questions. If relevant evidence is split across chunks, adjust boundaries or overlap rather than blindly increasing K.

Tune Chunk Size and Overlap

Chunk size and overlap are retrieval parameters, not universal constants. Start with a reasonable baseline and test alternatives on your own documents.

Overlap can preserve information at boundaries, but excessive overlap increases storage, retrieval duplicates, and context noise.

Improve Embeddings

Embeddings control how semantic similarity is represented. A better embedding model can improve retrieval when the existing model does not represent your domain or language well.

Use Metadata Filtering

Similarity alone may return a semantically related chunk from the wrong region, product, language, customer, or document version. Metadata filters can constrain the search space.

documents = [
    {"id": "a", "text": "Refunds are available for 30 days.", "region": "US"},
    {"id": "b", "text": "Refunds are available for 14 days.", "region": "EU"},
    {"id": "c", "text": "Refunds require the original receipt.", "region": "US"},
]

region = "US"
filtered = [doc for doc in documents if doc["region"] == region]

for doc in filtered:
    print(doc["id"], doc["text"])

This is a teaching example. A production vector database applies metadata filters during retrieval rather than scanning every document in Python.

Use Hybrid Search When Exact Terms Matter

Semantic retrieval is strong for meaning, while keyword retrieval can be strong for exact product names, IDs, error codes, policy numbers, and uncommon terms.

Hybrid search combines both signals. Evaluate whether the combination improves retrieval on the query types that matter for your application.

Retrieve Broadly, Then Rerank

A useful pattern is to retrieve a larger candidate set with a fast first-stage search and then use a stronger ranking method to put the best candidates first.

Reranking is especially useful when the initial retriever finds the right evidence but does not order it well. It also adds latency and compute cost, so measure the trade-off.

Do Not Solve Everything by Increasing Top-K

Increasing K can improve recall, but it can also increase irrelevant context, token usage, latency, and distraction.

Improve Context Construction

Retrieval results are not automatically good model context. Remove duplicates, preserve useful ordering, include source labels, and keep the final context focused.

Improve the RAG Prompt

The generation prompt should clearly define how retrieved context is used. A common pattern is to tell the model to answer from the supplied evidence and to acknowledge when the evidence is insufficient.

question = "What is the refund window?"
context = "Policy 17: Refunds are available for 30 days."

prompt = f"""
Answer the question using only the provided context.
If the context does not contain enough information, say that you do not have enough evidence.

Context:
{context}

Question:
{question}
"""

print(prompt)

The prompt improves grounding behavior, but it cannot create missing evidence. Retrieval quality still matters.

Control Hallucinations With Evidence Rules

Good RAG systems make unsupported answers less likely by defining what the model should do when evidence is missing or conflicting.

For conflicting documents, include version or effective-date metadata and define which source should take priority.

Improve Query Understanding

Some failures begin before retrieval because the user's wording is ambiguous, abbreviated, or conversational. Query rewriting can turn a short question into a clearer retrieval query.

Query rewriting should preserve the user's intent. An aggressive rewrite can introduce assumptions that were never requested.

Use a Better Retrieval Pipeline

This layered design is often more controllable than asking one retrieval step to solve every problem.

Improve With a Fixed Evaluation Set

Keep a representative dataset of questions and expected evidence. Every change to chunking, embeddings, retrieval, reranking, prompts, or models should be tested against the same baseline.

experiments = {
    "baseline": {"recall_at_5": 0.72, "answer_accuracy": 0.68},
    "hybrid": {"recall_at_5": 0.81, "answer_accuracy": 0.75},
    "hybrid_reranked": {"recall_at_5": 0.81, "answer_accuracy": 0.80},
}

for name, scores in experiments.items():
    print(name, scores)

Do not declare success from one impressive question. Look at aggregate metrics and inspect representative failures.

Measure More Than Accuracy

A production improvement should consider quality and system behavior together.

Common Improvement Mistakes

Practical Improvement Checklist

Mini Project: Improve a Weak RAG System

Imagine your baseline RAG system has Recall@5 = 0.65 and answer accuracy = 0.61. Your evaluation set shows that many failures come from wrong-region documents and poorly ranked candidates.

30-Second Recap

• Improve the layer that is actually failing.
• Start with clean, current source data.
• Tune chunking, embeddings, filters, retrieval, and reranking using evaluation data.
• Keep final context focused instead of blindly increasing Top-K.
• Use prompts and evidence rules to improve grounding, not to compensate for missing evidence.
• Measure quality together with latency and cost.
• Keep a fixed evaluation set so every change is comparable.

Quick Quiz

Key Takeaways

The most useful improvement is not necessarily a bigger model. Better data, chunking, retrieval constraints, ranking, and context construction can often make a larger difference.