Why Context Matters

The Same Question Can Have Different Answers

An LLM does not answer a question in isolation. The instructions, documents, conversation history, and other information you place in the request can change what the model should say.

Context Is the Model's Working Information

Think of the model as an analyst who needs the right evidence on the desk before making a decision. Your application decides what evidence to place there.

Important: Context is not just “more text.” It is selected information intended to help the model perform the current task.

Experiment: Change the Context

Here is a small experiment you can reproduce with an LLM API. The application asks the same question twice but supplies different company-policy context.

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
question = "Can I work from home on Friday?"

def ask(policy):
    response = client.responses.create(
        model="gpt-5-mini",
        input=[
            {"role": "system", "content": "Answer using only the supplied policy."},
            {"role": "user", "content": f"Policy:
{policy}

Question: {question}"},
        ],
    )
    return response.output_text

policy_a = "Remote work is allowed on Fridays."
policy_b = "Remote work is not allowed on Fridays."

print("Context A:", ask(policy_a))
print("Context B:", ask(policy_b))

What to observe: the user question does not change. The policy context changes, so the evidence available to the model changes. Keep API keys in environment variables and use the model/API version available in your account.

Relevant Context Beats Irrelevant Context

Suppose the user asks about parental leave. Adding the company's laptop policy, cafeteria menu, and office parking rules makes the prompt longer without making the answer better.

Context Quality Has Four Dimensions

Imagine two retrieved documents: one says “Returns accepted within 30 days,” while an old document says “Returns accepted within 14 days.” Simply putting both into the prompt can create ambiguity. Retrieval and application logic should prefer the current authoritative policy.

Build a Simple Context Builder

A context builder turns application data into a small, structured package for the model. Start simple: select the relevant records, label them, and add the user's question.

def build_context(question, policy, employee_type):
    return f"""Company Policy

Employee type: {employee_type}
Policy:
{policy}

Employee question:
{question}
"""

context = build_context(
    question="How many parental leave days do I have?",
    policy="Eligible employees receive 16 weeks of parental leave.",
    employee_type="full-time",
)

print(context)

In a production system, the policy value could come from a database, search result, or RAG retriever. The important design step is selecting useful evidence before calling the model.

Context Can Be Changed Deliberately

Changing context is not only a debugging technique. It is how many AI applications adapt to different users, products, policies, permissions, and tasks.

Key idea: the application controls context selection. The model does not automatically know which internal policy, database row, or private document your application wants it to use.

Break the Context on Purpose

A useful engineering exercise is to intentionally remove or corrupt the evidence and observe the result. Then restore the correct context.

This kind of failure test helps you distinguish a model problem from a context-selection problem. If the correct evidence is missing, changing the prompt wording alone may not solve the real issue.

Mini-Project: Company Policy Assistant

Build a small assistant that answers employee questions from a set of policy documents. Start with a few local Python strings, then replace the manual selection with retrieval as you learn RAG.

Context Is a Design Decision

Reliable LLM applications are not built by sending every piece of available information to the model. They are built by deciding what information the model needs for the task.