Token Limits
A model can only process a finite amount of tokenized information in one request. Learn how input, output, and context capacity interact—and how to design applications that stay within the limit.
By the end of this lesson, you will be able to:
- Explain what a token limit is and why it matters.
- Distinguish input tokens, output tokens, and the total context used by a request.
- Estimate whether a prompt plus expected response can fit within a model's capacity.
- Use Python to count tokens with a model-appropriate tokenizer.
- Apply practical strategies when content is too large for one request.
What Is a Token Limit?
A token limit is the maximum amount of tokenized context a model can handle for a request. The exact limit depends on the model and API, so it should never be assumed from another model.
Important: A token limit is not the same thing as a character limit or word limit. Tokenization determines how much text consumes the available capacity.
Input Tokens and Output Tokens
For a chat application, the model may receive much more than the latest user message. Previous messages, system instructions, retrieved documents, tool results, and other context can all consume input capacity.
What you send
Instructions, conversation history, documents, examples, and tool results become part of the model's input context.
What the model generates
The response also uses tokens. Reserving enough output space is important when the answer may be long.
A Practical Budget Example
Suppose a hypothetical model has a context capacity of 8,000 tokens. Your application sends 5,600 input tokens and asks for up to 1,500 output tokens.
This is a planning example, not a statement about any specific production model. In real applications, use the current model's documentation and tokenizer.
What Can Consume Your Context?
Token limits become easy to hit when an application keeps adding information to every request.
How Do You Know How Many Tokens You Have?
Do not estimate exact token counts from characters alone. Use the tokenizer associated with the model whenever the exact count matters.
from transformers import AutoTokenizer
# Use the tokenizer that matches the model family you are evaluating.
tokenizer = AutoTokenizer.from_pretrained("gpt2")
text = "Explain why token limits matter for an LLM application."
tokens = tokenizer.encode(text, add_special_tokens=False)
print("Token count:", len(tokens))
print("Token IDs:", tokens)Why this matters: the number is tokenizer-specific. The GPT-2 tokenizer above is a teaching tool for counting tokens; do not use its count as an exact count for a different model.
What Happens When You Exceed a Limit?
An application cannot simply keep adding tokens forever. Depending on the API and request, an oversized request may be rejected, truncated, or require you to reduce the context before sending it.
Too much context
Full chat history + every document + large tool output + long answer budget.
Managed context
Keep relevant history, retrieve only useful chunks, summarize older turns, and reserve output space.
Strategy 1: Trim or Summarize History
Long-running chat applications can accumulate thousands of tokens. Instead of sending every previous message, keep recent turns and summarize older information.
The goal is not to delete useful information blindly. Preserve facts the application actually needs and remove redundant history.
Strategy 2: Retrieve Only Relevant Context
RAG systems should not paste an entire knowledge base into the prompt. Retrieval narrows a large collection to a smaller set of relevant chunks.
Strategy 3: Control the Output Budget
If your request uses most of the available capacity for input, there may not be enough room for a long response. Set a sensible output limit for the task.
When a response genuinely needs to be long, consider generating it in stages rather than forcing one request to do everything.
Real Application: Context Budgeting
Imagine a company assistant with a model capacity of 16,000 tokens. Your request contains 2,000 tokens of instructions, 4,000 tokens of recent conversation, and 7,000 tokens of retrieved documents. You reserve 2,000 tokens for the answer.
Now imagine retrieval grows to 10,000 tokens. The same request would plan for 18,000 tokens and no longer fit. A good application notices this before sending the request and reduces, summarizes, or reranks the context.
Build a Token Budget Checker
Write a small Python function that accepts a context capacity, input token count, and output budget. Return whether the planned request fits and how much headroom remains.
def check_budget(capacity, input_tokens, output_budget):
planned = input_tokens + output_budget
remaining = capacity - planned
return planned <= capacity, remaining
fits, remaining = check_budget(8000, 5600, 1500)
print(fits, remaining)Test yourself
Which strategy is most appropriate when a RAG application retrieves far more text than the model needs?
30-Second Recap
- Token limits describe how much tokenized context a model can handle for a request.
- Input includes more than the latest user message: history, instructions, retrieved text, and tool results can all consume tokens.
- Output tokens also need room, so budget for the response.
- Exact token counts depend on the tokenizer; use the model-appropriate tokenizer when precision matters.
- When context is too large, trim history, summarize, retrieve fewer relevant chunks, rerank, or split the task into multiple steps.