AI Chatbot

What Are We Building?

Our project is a simple support-style chatbot. A user sends a message, the application sends the relevant conversation context to an LLM, and the assistant returns a helpful answer.

Project Architecture

Keep the chatbot architecture simple at first. Add RAG, tools, or agents only when the product actually needs them.

Choose the Chatbot Scope

Define what the bot should and should not do before writing code. A narrow scope makes prompting, evaluation, and safety much easier.

• Answer common product questions.
• Explain policies in plain language.
• Ask for clarification when context is missing.
• Admit when it does not know an answer.
• Do not invent account or order information.
• Escalate requests that require a human.

Prepare Your Python Environment

The examples use Python. In a real project, keep secrets such as API keys outside source code.

import os

API_KEY = os.environ.get("OPENAI_API_KEY")

if not API_KEY:
    raise RuntimeError("OPENAI_API_KEY is not configured")

print("Environment is ready")

For production, use your platform's secret manager rather than committing credentials to Git.

Design the System Prompt

The system prompt defines the assistant's role, boundaries, tone, and response behavior. Keep stable instructions separate from user messages.

Represent Conversation Messages

Most chat APIs represent a conversation as ordered messages with roles. Keep the structure explicit so it can be validated and trimmed.

messages = [
    {"role": "system", "content": "You are a helpful support assistant."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Open Settings and choose Security."},
    {"role": "user", "content": "What if I cannot access my email?"},
]

print(len(messages))

Make the Model Call

The exact SDK differs by provider, but the application pattern is the same: send instructions and conversation context, then handle the model response.

def build_request(messages: list[dict]) -> dict:
    return {
        "model": "your-model",
        "messages": messages,
        "temperature": 0.2,
    }

request = build_request(messages)
print(request["model"])

Treat the provider SDK call as an implementation detail behind a small application service.

Control Conversation Memory

Sending the entire conversation forever increases tokens, latency, and cost. Start with a recent-message window and add summarization when conversations become long.

Trim Context with Python

A basic sliding window is often enough for an initial chatbot.

def keep_recent_messages(messages: list[dict], max_messages: int = 8):
    system = messages[:1]
    recent = messages[-max_messages:]
    return system + recent

trimmed = keep_recent_messages(messages, max_messages=4)
print(len(trimmed))

Add a Chat History Store

For a real application, conversation state normally lives outside the request process. Store messages by a conversation ID and user or tenant ID.

Streaming Improves Perceived Speed

Instead of waiting for the complete response, stream generated tokens or chunks to the UI. This can make the chatbot feel much faster even when total generation time is unchanged.

Handle Errors Gracefully

Model providers can return timeouts, rate limits, authentication failures, or temporary service errors. Never expose raw stack traces or provider secrets to users.

• Set a request timeout.
• Retry only transient failures.
• Use exponential backoff.
• Limit retry attempts.
• Return a friendly fallback message.
• Log a correlation ID for debugging.

Protect Against Prompt Injection

User input is untrusted data. Do not let a user message silently override system policies or reveal hidden instructions.

Protect Sensitive Data

A chatbot may process names, emails, account details, internal documents, or other sensitive information. Minimize what reaches the model and enforce authorization before retrieving private data.

When Should You Add RAG?

Add retrieval when the chatbot must answer from a knowledge base that changes over time or is not part of the model's reliable knowledge.

When Should You Add Tools?

Tools are useful when the chatbot must obtain live information or perform an action. The model should propose the tool call; your application validates and executes it.

Add Guardrails

Guardrails should cover both inputs and outputs. Keep the rules close to the application boundary instead of relying only on the prompt.

• Validate message length and format.
• Block or review risky requests.
• Protect system instructions and secrets.
• Restrict tool permissions.
• Validate structured outputs.
• Provide a human escalation path.

Monitor the Chatbot

Track enough telemetry to understand failures without storing more user content than necessary.

Evaluate the Chatbot

Create a small golden test set before launch. Include normal questions, edge cases, known failures, safety cases, and questions that require the bot to admit uncertainty.

Complete Project Flow

Put the pieces together into a simple request lifecycle.

Mini Project: Support Chatbot

Build a support chatbot that answers product questions and knows when to escalate.

Production Checklist