Building Basic RAG

What We Are Building

A basic RAG application has two connected flows. First, we prepare knowledge. Then, when a question arrives, we retrieve relevant knowledge and give it to an LLM.

Step 1: Start With Small Documents

For learning, use a short list of documents instead of beginning with PDFs or a large database. Each document should contain information that can answer a realistic question.

These examples are intentionally small. The same architecture can later work with larger document collections.

Step 2: Represent the Knowledge

Store each piece of knowledge as text plus optional metadata. Metadata helps you trace a result back to its source and can later support filtering.

Step 3: Chunk the Documents

Real documents are usually too large to treat as one retrieval unit. Split them into chunks that preserve useful meaning.

text = "Refunds are allowed within 30 days. Shipping takes 3 to 5 business days."

# Very simple teaching example: split into sentences.
chunks = [sentence.strip() for sentence in text.split(".") if sentence.strip()]

for chunk in chunks:
    print(chunk)

This is not a production-grade chunker. It is a simple way to see where chunking fits in the pipeline.

Step 4: Create Embeddings

An embedding model converts each chunk into a vector. Similar meanings should produce vectors that are useful for similarity search.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

chunks = [
    "Refunds are allowed within 30 days.",
    "Standard shipping takes 3 to 5 business days.",
    "Business plans include priority support."
]

vectors = model.encode(chunks)
print(vectors.shape)

The exact embedding model is a design choice. What matters here is the flow: text → embedding vectors → similarity search.

Step 5: Store the Vectors

A basic prototype can keep vectors in memory. A real application will usually use a vector index or vector database when the collection becomes larger or needs persistence.

Step 6: Create the User Query

When the user asks a question, embed the question using the same embedding space used for the stored chunks.

Step 7: Retrieve Relevant Chunks

Compare the query vector with stored vectors and select the highest-scoring chunks. This is the retrieval step.

from sentence_transformers import SentenceTransformer
from numpy import dot
from numpy.linalg import norm

model = SentenceTransformer("all-MiniLM-L6-v2")
chunks = [
    "Refunds are allowed within 30 days.",
    "Standard shipping takes 3 to 5 business days.",
    "Business plans include priority support."
]

chunk_vectors = model.encode(chunks)
query = "How long do I have to request a refund?"
query_vector = model.encode([query])[0]

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))

scores = [cosine(query_vector, vector) for vector in chunk_vectors]
ranked = sorted(zip(chunks, scores), key=lambda item: item[1], reverse=True)

for text, score in ranked:
    print(round(score, 3), text)

The highest-scoring result should be the refund chunk for this simple example. In a production system, use an appropriate vector index or vector database rather than scanning every vector manually.

Step 8: Choose Top-K

Usually, you do not send every retrieved chunk to the LLM. Top-K controls how many candidates are selected for the next stage.

There is no universally correct K. Choose it using retrieval evaluation and the type of questions your application receives.

Step 9: Build the Context

The retrieved chunks become evidence for the LLM. A simple context builder can label each source and join the selected text.

retrieved = [
    {"source": "refund-policy", "text": "Refunds are allowed within 30 days."},
    {"source": "shipping-policy", "text": "Standard shipping takes 3 to 5 business days."}
]

context = "\n\n".join(
    f"Source: {item['source']}\n{item['text']}"
    for item in retrieved
)

print(context)

Good context is relevant, concise, and traceable. Avoid dumping every available document into the prompt.

Step 10: Augment the Prompt

RAG adds the retrieved evidence to the user question before sending the request to the generation model.

Step 11: Generate a Grounded Answer

The LLM receives the question plus selected context. A strong prompt should tell the model to use the supplied evidence and avoid inventing facts that are not supported.

question = "How long do I have to request a refund?"
context = "Refunds are allowed within 30 days."

prompt = f"""Answer the question using only the context below.
If the context does not contain the answer, say that you do not have enough information.

Context:
{context}

Question:
{question}
"""

print(prompt)

This example prepares the prompt but does not call a paid or hosted LLM. An application would send this prompt to its selected generation model.

Step 12: Return the Answer and Source

A useful RAG application should preserve source information so the application can show where the evidence came from.

Source labels or citations should correspond to the actual retrieved evidence. A citation by itself does not make an unsupported answer correct.

Putting the Basic RAG Loop Together

Here is the complete teaching model in one place.

A Compact Python Prototype

This prototype demonstrates the retrieval portion of a basic RAG system. It intentionally keeps the index in memory so the logic is easy to understand.

from sentence_transformers import SentenceTransformer
from numpy import dot
from numpy.linalg import norm

model = SentenceTransformer("all-MiniLM-L6-v2")

knowledge = [
    {"source": "refund-policy", "text": "Refunds are allowed within 30 days."},
    {"source": "shipping-policy", "text": "Standard shipping takes 3 to 5 business days."},
    {"source": "support-policy", "text": "Business plans include priority support."},
]

vectors = model.encode([item["text"] for item in knowledge])
query = "How long can I request a refund?"
query_vector = model.encode([query])[0]

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))

ranked = sorted(
    zip(knowledge, vectors),
    key=lambda item: cosine(query_vector, item[1]),
    reverse=True,
)

top_k = ranked[:2]
context = "\n\n".join(item[0]["text"] for item in top_k)

print("Retrieved context:")
print(context)

To turn this into a complete application, connect the selected context to an LLM API and return the generated answer with the retrieved source metadata.

What This Prototype Simplifies

Basic RAG vs Production RAG

The prototype teaches the architecture. Production engineering adds reliability, scale, security, observability, and continuous evaluation.

When Basic RAG Works Well

Common Mistakes

How to Improve Your Basic RAG