RAG Pipeline

The RAG Pipeline at a Glance

A Retrieval-Augmented Generation pipeline connects a user's question to a knowledge source before an LLM generates the final answer.

Stage 1: Collect the Knowledge

Before users ask questions, the system needs a reliable knowledge source. Sources can include PDFs, web pages, product documentation, databases, support articles, or internal documents.

Stage 2: Load and Clean Documents

Raw files are rarely ready for retrieval. An ingestion process extracts text, removes unnecessary noise, and preserves useful structure and metadata.

Stage 3: Chunk the Content

Large documents are divided into smaller retrieval units called chunks. Chunking makes it easier to retrieve only the part that matters to a question.

Chunk size and boundaries affect retrieval quality, so chunking is an important part of the pipeline.

Stage 4: Create Embeddings

Each chunk can be converted into a numerical vector called an embedding. The vector represents useful semantic information about the text.

The same embedding model or a compatible strategy is normally used for user queries so their vectors can be compared with stored chunk vectors.

Stage 5: Store Vectors and Metadata

The vectors are stored in a vector index or vector database along with the information needed to identify and filter the original content.

Stage 6: Receive the User Query

When a user asks a question, the online part of the RAG pipeline begins. The application first receives and validates the query.

Stage 7: Embed the Query

The query can be converted into an embedding using an embedding model. This creates a vector that can be compared with stored document vectors.

Stage 8: Retrieve Candidate Chunks

The retriever searches the vector index and returns a candidate set of relevant chunks. The system often retrieves more candidates than it finally sends to the LLM.

This is where concepts such as Top-K, metadata filtering, hybrid search, and similarity search become useful.

Stage 9: Filter and Rerank When Needed

Retrieval can be improved by applying metadata filters or a second-stage reranker. These steps help control which candidates are considered and which evidence is strongest.

Stage 10: Build the Context

The selected chunks are assembled into a context that the LLM can understand. Good context is relevant, clearly separated, and includes useful source information when appropriate.

retrieved_chunks = [
    "Refunds are allowed within 30 days of purchase.",
    "Refund requests can be submitted through the support portal."
]

context = "\n\n".join(retrieved_chunks)

print(context)

This example only demonstrates context construction. A production retriever would obtain the chunks from a search system.

Stage 11: Augment the Prompt

RAG adds the retrieved context to the model input. The prompt should clearly tell the model how to use the supplied evidence and what to do when the evidence is insufficient.

question = "How long do I have to request a refund?"
context = "Refunds are allowed within 30 days of purchase."

prompt = f"""
Answer the question using the context below.
If the context does not contain the answer, say that
there is not enough information.

Context:
{context}

Question:
{question}
"""

print(prompt)

Stage 12: Generate the Answer

The LLM receives the question plus the selected context and generates the response. The generator is not responsible for finding the documents; retrieval has already done that job.

Stage 13: Return Sources and Citations

Many RAG applications return source references along with the answer. This can help users inspect the evidence and can make the system easier to audit.

Citations do not automatically prove that an answer is correct. They are useful only when they accurately correspond to the evidence used.

Offline vs Online Parts of RAG

It is useful to divide the architecture into ingestion time and query time.

Complete RAG Pipeline

Simple End-to-End Python Mental Model

The following code models the flow without connecting to a real vector database or LLM API.

documents = [
    "Refunds are allowed within 30 days.",
    "Standard shipping takes 3 to 5 business days.",
    "Premium support is available on business plans."
]

question = "How long is the refund period?"

# Simplified retrieval for learning purposes.
retrieved = [documents[0]]

context = "\n".join(retrieved)
answer = f"Using the retrieved context: {context}"

print(answer)

The real pipeline replaces the simplified retrieval step with embeddings, vector search, filtering, reranking, and an actual LLM call.

Where the Pipeline Can Fail

How to Improve a RAG Pipeline

RAG Evaluation: Measure Every Stage

Evaluating only the final answer can hide retrieval problems. A useful RAG evaluation strategy measures multiple stages.