RAG Application

What Are We Building?

Build an RAG application that answers employee questions using company documents, policies, product information, and knowledge bases. The assistant should retrieve relevant retrieved knowledge before generating an answer.

Understand the Architecture

A practical architecture separates ingestion from question answering: documents are collected and indexed once, while employee questions trigger retrieval and generation.

Connect Knowledge Sources

Typical sources include PDFs, DOCX files, web pages, support articles, databases, and application data. Choose sources that are trustworthy, current, and relevant to the use case.

Ingest Documents

The ingestion pipeline loads source content, extracts text, preserves metadata, cleans noise, chunks the content, creates embeddings, and stores searchable records.

Preserve Metadata

Keep document ID, title, department, source URL, access group, section, timestamps, and version information. Metadata becomes important for filtering, citations, and accesss.

Chunk Company Knowledge

Good chunks preserve a coherent policy, procedure, product explanation, or section. Avoid chunks that mix unrelated departments or topics.

Create Embeddings

Embeddings turn chunks into vectors so a question such as “How many days can I work remotely?” can retrieve semantically related policy text even when the wording differs.

from dataclasses import dataclass

@dataclass
class CompanyChunk:
    text: str
    document_id: str
    department: str
    access_group: str

chunk = CompanyChunk(
    text="Employees receive 20 paid vacation days.",
    document_id="hr-policy-2026",
    department="hr",
    access_group="employees",
)

print(chunk.document_id, chunk.department)

Store the Knowledge Index

Use a vector database or search index to store embeddings, text, and metadata. Keep the original source reference so answers can show where information came from.

Retrieve Relevant Knowledge

At query time, embed the employee question, retrieve candidate chunks, apply metadata and access filters, and optionally rerank the candidates before generation.

Apply Metadata and Access Filters

Retrieval must respect the employee’s accesss. A user should never receive retrieved text from a private HR, finance, legal, or executive document they are not authorized to access.

Build a Grounded Prompt

Give the model only the selected company context and clear instructions to answer from that context. If the evidence is insufficient, the assistant should say that it cannot verify the answer.

def build_grounded_prompt(question: str, contexts: list[str]) -> str:
    evidence = "\n\n".join(contexts)
    return f"""Answer using only the company evidence below.
If the evidence is insufficient, say you cannot verify the answer.

Evidence:
{evidence}

Question: {question}"""

prompt = build_grounded_prompt(
    "How many vacation days do employees receive?",
    ["HR policy: Employees receive 20 paid vacation days."],
)
print(prompt)

Return Sources and Citations

Show document titles, sections, page numbers, or links where possible. Citations let employees verify important answers and help administrators debug retrieval.

Handle Unknown Questions

Do not force an answer. If retrieval confidence is low or the context does not support the question, route to a human, ask for clarification, or state that the knowledge base does not contain enough information.

Keep Knowledge Fresh

Company information changes. Track document versions, update timestamps, ingestion status, and deletion events. Re-index changed content instead of blindly creating duplicates.

• Track source version
• Record updated timestamp
• Remove deleted content
• Re-index changed documents

Combine RAG with Tools

Use RAG for knowledge and tools for live actions. For example, retrieve the leave policy with RAG, then call an HR system to check an employee’s remaining leave balance.

Monitor the Assistant

Track retrieval quality, answer quality, latency, token usage, citations, failed searches, access denials, and user feedback.

def can_access(user_groups: set[str], required_group: str) -> bool:
    return required_group in user_groups

user_groups = {"employees", "engineering"}
print(can_access(user_groups, "employees"))
print(can_access(user_groups, "executives"))

Evaluate with Real Questions

Create a representative evaluation set from real employee questions. Measure retrieval relevance, groundedness, citation correctness, refusal behavior, and answer usefulness.

Secure the Knowledge Base

Protect source documents, embeddings, credentials, logs, and prompts. Treat retrieved company data as sensitive application data and apply least-privilege access.

Build the RAG Application

A useful first version can support one department, a small document set, citations, access filters, feedback buttons, and an escalation path. Expand after measuring quality.

Common Mistakes

Indexing everything without ownership, ignoring accesss, returning uncited answers, using stale documents, retrieving too much context, and treating the LLM as the source of truth are common production mistakes.

Mini Project: RAG Question Answering Assistant

Create an assistant for a product, support, or internal knowledge domain. Index approved documents, attach metadata, filter by employee access, retrieve the top candidates, generate a grounded answer, and return citations plus a “source or human fallback” fallback.

30-Second Recap

RAG Application = trusted sources + ingestion + chunking + embeddings + retrieval + access filtering + grounded generation + citations + monitoring.

Key Takeaways

Treat retrieved knowledge as a governed data system. Retrieval should be access-aware, answers should be grounded and cited, and freshness and evaluation must be part of the design.