LESSON 15 Β· TOKENS & CONTEXT

Tokenization

Tokenization is the step that turns text into the pieces an LLM can process. Learn how tokenizers split text, assign token IDs, and handle words, punctuation, numbers, and unfamiliar words.

14 min readβ€’Beginnerβ€’Generative AI

By the end of this lesson, you will be able to:

  • Explain what tokenization does and why LLMs need it.
  • Distinguish characters, words, tokens, and token IDs.
  • Understand why subword tokenization is useful.
  • Inspect real tokenization with Python and a tokenizer library.
  • Explain why different models can tokenize the same text differently.
1

What Is Tokenization?

Tokenization is the process of converting text into a sequence of tokens. A tokenizer applies a vocabulary and rules to decide which pieces of text become tokens.

Simple idea:Before an LLM can process β€œI love Python!”, the text must be converted into token pieces and then into numeric token IDs.
TextI love Python!
β†’
Token piecesPieces selected by tokenizer
β†’
Token IDsNumbers from vocabulary
β†’
ModelProcesses the sequence
2

Words Are Not the Unit the Model Must Use

A tokenizer does not have to split text only at spaces. It may keep common words together while splitting less common words into reusable subword pieces.

WORD

helpful

A common word might be represented as one token or a small number of pieces.

SUBWORD

unhelpful

A tokenizer can reuse pieces related to un, help, and ful rather than needing every possible word in its vocabulary.

PUNCTUATION

Hello!

The exclamation mark is text too. It can receive its own token or be grouped according to the tokenizer.

Important: These displayed pieces are teaching examples, not a claim about one specific model's exact vocabulary. Exact tokenization depends on the tokenizer.
3

Why Use Subword Tokenization?

A vocabulary containing every possible word would be enormous. Subword tokenization lets a model reuse pieces across many words.

Word-only approach

β€œunhappiness” is treated as a single vocabulary item.

huge vocabulary pressure
β†’

Subword approach

The tokenizer can compose unfamiliar words from reusable pieces.

reusable vocabulary

This helps balance vocabulary size, sequence length, and the ability to represent uncommon words. It is one reason modern LLM tokenizers are not simply dictionaries of whole words.

4

Tokenization Happens Before the Model Sees the Numbers

1. Raw textβ€œAI learns from data.”
2. TokenizerApplies its vocabulary and tokenization algorithm.
3. TokensText pieces represented by the vocabulary.
4. Token IDsEach vocabulary entry has an integer ID.
5. Model inputIDs are mapped into learned numerical representations.
5

One Sentence Can Contain Different Kinds of Tokens

Consider:

Order #4821 is ready!
Order#4821isready!

This row is illustrative. A real tokenizer may split 4821, attach spaces to neighboring pieces, or use other boundaries. The lesson is that tokenization is algorithm- and vocabulary-dependent.

Words

Human-friendly units.

Punctuation

Can be tokenized too.

Numbers

May become one or several tokens.

Spaces

May affect token boundaries depending on tokenizer.

6

Different Tokenizers Can Produce Different Results

There is no universal tokenization of English text. Two models can use different vocabularies and algorithms, so the same sentence can produce different token IDs and token counts.

Tokenizer A

Vocabulary and rules A

tokens β†’ IDs

Tokenizer B

Vocabulary and rules B

possibly different tokens β†’ different IDs
Engineering rule:Always use the tokenizer associated with the model or API when exact token counts matter.
7

See Tokenization With Python

For a real tokenizer experiment, you can use a Hugging Face tokenizer. The code below loads a tokenizer and prints the tokens and IDs. The exact output depends on the selected model tokenizer.

</> Python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "distilbert/distilbert-base-uncased"
)

text = "Tokenization helps an LLM process text."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text, add_special_tokens=False)

print("Tokens:", tokens)
print("Token IDs:", token_ids)
print("Token count:", len(token_ids))
What to observe: Run the code and inspect which pieces are preserved as whole words and which are split. Do not memorize the exact IDs; they belong to this tokenizer's vocabulary.
8

Change the Text and Experiment

Now change only the input string and compare the token count.

playCompareshort, common word
playingComparerelated form
unpredictableComparelonger word
Machine learning is useful.Comparesentence + punctuation

Ask: Which inputs are split into more pieces? Which words are reused as recognizable pieces? This is a practical way to build intuition instead of relying on the rule β€œone word equals one token.”

9

Tokenization and Multilingual Text

Token counts can vary significantly across languages and writing systems. A tokenizer's vocabulary is designed around the data used to build it, so different languages may require different numbers of tokens for comparable text.

EnglishHow are you?
ζ—₯本θͺžγŠε…ƒζ°—γ§γ™γ‹οΌŸ
Ψ§Ω„ΨΉΨ±Ψ¨ΩŠΨ©ΩƒΩŠΩ Ψ­Ψ§Ω„ΩƒΨŸ

Do not assume an English word-to-token estimate works equally well for every language. Measure with the tokenizer you actually use.

10

Common Tokenization Mistakes

β€œEvery word is one token.”

False. Words can be split into multiple pieces.

β€œSpaces are always separate tokens.”

False. Token boundaries depend on the tokenizer.

β€œAll LLMs use the same tokenizer.”

False. Tokenizers and vocabularies differ.

β€œToken IDs have universal meaning.”

False. An ID is meaningful within its tokenizer's vocabulary.

PRACTICE

Build a Tokenization Explorer

Use the Python example above. Test at least five inputs: a short word, a long word, punctuation, a number, and text in another language. Record the token pieces and token count.

☐ Print tokens☐ Print token IDs☐ Compare counts☐ Try another language
Open Google Colab β†’
QUICK QUIZ

Why do modern tokenizers often use subword pieces?

30-second recap

  • Tokenization converts text into model-readable token pieces.
  • Tokens are mapped to vocabulary IDs before entering the model.
  • Subword pieces help represent common and uncommon words with a reusable vocabulary.
  • Punctuation, numbers, spaces, and multilingual text can affect tokenization.
  • Different models can tokenize the same text differently.
  • When exact counts matter, use the tokenizer associated with the model.