RAG with Multiple Documents

Why Multiple Documents?

Real knowledge assistants rarely depend on one file. A company may have hundreds of policies, product manuals, FAQs, contracts, technical guides, and reports. The user's question may require information from one document or several documents.

Single-Document vs Multi-Document RAG

The retrieval problem becomes more interesting as the collection grows: irrelevant documents, duplicate content, outdated versions, and permission boundaries can all affect the answer.

The Multi-Document RAG Pipeline

Separate the system into ingestion time and query time. Ingestion happens when documents enter or change in the knowledge base. Query time happens for every user question.

Document Identity Is Essential

When thousands of chunks share one vector collection, you still need to know exactly where every chunk came from.

document = {
    "document_id": "travel-policy-2026",
    "title": "Domestic Travel Policy",
    "source": "policies/travel-policy-2026.pdf",
    "version": "2026.1",
    "department": "finance",
}

chunk = {
    "text": "Domestic travel reimbursement is limited to ...",
    "document_id": document["document_id"],
    "title": document["title"],
    "version": document["version"],
    "department": document["department"],
    "page": 7,
}

print(chunk["document_id"], chunk["page"])

Choose a Metadata Schema

Metadata lets you filter, group, debug, cite, and reason about documents before and after vector search.

Process Many Documents

A simple ingestion loop can process a collection one document at a time. Production systems usually add queues, retries, change detection, and batch processing.

documents = [
    {"id": "policy-01", "path": "policies/leave.txt"},
    {"id": "policy-02", "path": "policies/travel.txt"},
    {"id": "manual-01", "path": "manuals/product.txt"},
]

records = []

for document in documents:
    with open(document["path"], "r", encoding="utf-8") as file:
        text = file.read()

    words = text.split()
    for start in range(0, len(words), 300):
        chunk = " ".join(words[start:start + 300])
        if chunk:
            records.append({
                "document_id": document["id"],
                "text": chunk,
            })

print("Documents:", len(documents))
print("Chunks:", len(records))

This example demonstrates the data flow. A production ingestion service would also create embeddings and persist the records in a vector database.

One Collection or Separate Collections?

There is no universal rule. Choose the design based on tenant isolation, document relationships, scale, operational complexity, and retrieval requirements.

Embed Chunks from Every Document

All chunks that need to be compared in the same vector search should use a compatible embedding representation.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

texts = [record["text"] for record in records]
vectors = model.encode(texts, normalize_embeddings=True)

print("Chunks:", len(texts))
print("Vector dimensions:", vectors.shape[1])

The model name is a teaching example. In a real application, select an embedding model appropriate for your language, quality, latency, and infrastructure requirements.

Retrieve Across the Whole Collection

A user's question should be compared against the indexed chunks from all documents the user is allowed to access.

import numpy as np

question = "What is the domestic travel reimbursement limit?"
query_vector = model.encode(question, normalize_embeddings=True)

scores = vectors @ query_vector
best_indices = np.argsort(scores)[::-1][:5]

for index in best_indices:
    record = records[index]
    print(round(float(scores[index]), 3), record["document_id"])
    print(record["text"][:180], "\n")

Use Metadata Filtering Before Retrieval

Do not search every document if the application already knows a useful constraint. Filtering can reduce noise and prevent unauthorized sources from entering the candidate set.

allowed = [
    record for record in records
    if record.get("department") in {"finance", "hr"}
]

print("Authorized candidates:", len(allowed))

In a real vector database, apply supported metadata filters as part of the search request rather than loading the entire collection into application memory.

Top-K Can Return Duplicate Sources

Suppose the top five chunks all come from the same document. That may be correct, but it can also leave useful evidence from another relevant document out of the context.

Rerank the Candidates

For larger collections, a first-stage vector search can retrieve a candidate set and a reranker can reorder those candidates using the full query and passage together.

This two-stage approach balances broad recall with more precise final ranking.

Build Context Without Losing Sources

Before sending evidence to an LLM, format each selected chunk with enough metadata to keep its origin visible.

selected = [
    {
        "title": "Domestic Travel Policy",
        "document_id": "travel-policy-2026",
        "page": 7,
        "text": "Domestic travel reimbursement is limited to ...",
    },
    {
        "title": "Expense FAQ",
        "document_id": "expense-faq",
        "page": 3,
        "text": "Receipts are required for claims above ...",
    },
]

context = "\n\n".join(
    f"Source: {item['title']} | page {item['page']}\n{item['text']}"
    for item in selected
)

print(context)

When One Question Needs Multiple Documents

Some questions are naturally cross-document questions. The system should allow evidence from multiple sources when the task requires it.

Handle Versions and Duplicates

Multiple documents often contain the same information. Old versions can be especially dangerous because they may look highly relevant to the query.

Access Control in Multi-Document RAG

A retrieval system must not expose content merely because it is semantically relevant. Authorization is a separate requirement from relevance.

For multi-tenant systems, treat tenant or access scope as a hard retrieval boundary rather than a soft ranking preference.

Document-Level and Chunk-Level Retrieval

Sometimes the first question is not “which chunk?” but “which documents are relevant?” A system can use a two-step approach: discover useful documents, then retrieve precise chunks inside them.

This can be useful when documents have strong identities, such as product manuals, case files, projects, or policy collections.

Cross-Document Context Assembly

More retrieved chunks do not automatically mean a better answer. Select evidence that is relevant, non-redundant, recent, authorized, and small enough to fit the context budget.

Common Multi-Document RAG Problems

Evaluate Retrieval Across the Collection

Test more than whether the final answer sounds good. Build questions whose correct evidence is known and measure whether the right documents and chunks are retrieved.

A Simple End-to-End Mental Model

def multi_document_rag(question, documents):
    # 1. Keep only documents the user may access.
    authorized = [
        doc for doc in documents
        if doc["access"] == "allowed"
    ]

    # 2. Retrieve relevant chunks across the collection.
    candidates = retrieve(question, authorized)

    # 3. Improve ranking and remove redundant evidence.
    selected = rerank_and_select(question, candidates)

    # 4. Build source-aware context.
    context = "\n\n".join(
        f"[{item['title']} | page {item['page']}] {item['text']}"
        for item in selected
    )

    # 5. Generate only from the supplied evidence.
    return generate_answer(question, context)

When Multi-Document RAG Works Well

Production Checklist