What are Tokens?
Learn how text is broken into tokens, why LLMs work with tokens instead of plain words, and how tokens affect context, cost, and model behavior.
By the end of this lesson, you will be able to:
- Explain what a token is in simple terms.
- Understand why one word does not always equal one token.
- See how words, punctuation, numbers, and subwords can become tokens.
- Estimate token counts and understand why they matter for LLMs.
- Use Python to inspect token-like pieces with a simple tokenizer.
- Connect tokens to context windows, API cost, latency, and model input/output.
So, What Exactly Is a Token?
A token is a piece of text that an AI model processes. Depending on the tokenizer, a token can represent a whole word, part of a word, punctuation, or another piece of text.
One Word Does Not Always Mean One Token
Tokenizers do not simply split every sentence at spaces. They use a vocabulary and tokenization rules to divide text into reusable pieces.
Common short word
A word such as cat may be represented by one token.
Long or uncommon word
A word such as unhappiness can be split into multiple pieces.
Real-World Example: A Customer Message
Imagine an AI support application receives:
The tokenizer breaks the message into smaller pieces that can be represented numerically.
Notice that the order number can be split into multiple tokens. Punctuation can also be represented separately. That is why counting words is not the same as counting tokens.
What Can Become a Token?
Depending on the tokenizer, tokens may represent several kinds of text pieces.
hello
ing, tion
!, ?, .
2026 or pieces of a number
#, @, and other symbols
Whitespace or byte-level pieces may influence boundaries.
Why Do LLMs Use Tokens?
Human language contains an enormous number of words, names, spellings, numbers, and symbols. A token vocabulary gives the model a practical set of reusable pieces.
Reusable pieces
The same token can appear in many different words and sentences.
Manageable vocabulary
The model works with a finite vocabulary rather than treating every possible word as unique.
Numerical input
Tokens become IDs and numerical representations that neural networks can process.
From Text to Model Input
Tokenization is one stage in a larger pipeline:
- Text: You type a message.
- Tokenization: The tokenizer splits the text into token pieces.
- Token IDs: Each token maps to an integer ID from the vocabulary.
- Embeddings: IDs are mapped to numerical vectors the neural network can process.
- Model: The LLM uses the representation to predict the next token.
Python Example: See Token-Like Pieces
We can build a tiny educational tokenizer to make the idea concrete. This is not the tokenizer used by a production LLM.
import re
text = "I love Generative AI!"
# Educational tokenizer: words + punctuation
tokens = re.findall(r"\w+|[^\w\s]", text)
print("Tokens:", tokens)
print("Token count:", len(tokens))Tokens: ['I', 'love', 'Generative', 'AI', '!']A real LLM tokenizer can split these pieces differently. The goal is to understand text → pieces → count.
Why Token Count Matters
Modern LLM systems measure several limits and resources in tokens rather than words.
Context
Your prompt, conversation history, retrieved documents, and generated output all consume context tokens.
Cost
Many model APIs price usage using input and output tokens, so more tokens can increase cost.
Latency
Larger inputs and longer outputs generally require more computation and can affect response time.
Limits
Applications may need to shorten, summarize, chunk, or otherwise manage text when limits are reached.
Words vs Tokens: A Useful Mental Model
Words
Useful for humans when describing document size.
Tokens
Useful for understanding how an LLM receives and processes text.
There is no universal exact conversion such as “1 word = 1 token.” The ratio depends on language, vocabulary, text style, and tokenizer. Treat simple word-to-token conversions as estimates, not rules.
Common Mistakes Beginners Make
False. Words can be split into multiple tokens, and punctuation can also be tokenized.
False. Models can use different tokenizers and vocabularies.
False. Extra text can increase cost and context pressure without improving the answer.
False. Token count measures text pieces, not usefulness or correctness.
Build a Tiny Token Counter
Test at least five sentences. Try punctuation, a long word, a number, and symbols. Compare the number of words with the number of token-like pieces.
import re
sentences = [
"AI is useful.",
"Can you help me?",
"unhappiness",
"Order #4821",
"AI + Python = useful!",
]
for sentence in sentences:
tokens = re.findall(r"\w+|[^\w\s]", sentence)
words = sentence.split()
print(sentence)
print("words =", len(words), "tokens =", len(tokens))
print()Which statement about tokens is correct?
30-second recap
- A token is a piece of text an LLM processes.
- A token can be a whole word, part of a word, punctuation, or another text piece.
- One word does not necessarily equal one token.
- Tokens become IDs and numerical representations for the model.
- Token counts matter for context, cost, latency, and limits.
- Exact tokenization depends on the tokenizer and model.