Choosing Chunk Size

What Is Chunk Size?

Chunk size is the amount of content placed into one chunk before that chunk is embedded and stored for retrieval. Depending on the chunker, size can be measured in characters, words, tokens, sentences, or another unit.

The important question is not “What number should I always use?” It is “How much context does one retrieved chunk need to answer the questions users actually ask?”

Why Chunk Size Matters

A useful chunk size sits between these extremes: enough context to preserve meaning, but small enough for precise retrieval.

Small vs Large Chunks

Tokens Are Often a Better Unit

Words and characters are convenient, but tokens are often more useful when working with language models because model context limits are expressed in tokens.

text = "Chunk size controls how much context each retrieved unit contains."

words = text.split()
print("Words:", len(words))
print("Approximate chunk target:", 12, "words")

This is only a simple illustration. Production systems should use a tokenizer appropriate for the embedding or generation model when token limits matter.

Start With the Document Structure

Chunk size should not be chosen independently of the document. A technical manual, FAQ, legal policy, and product catalog have different natural units of meaning.

Chunk Size and Overlap

Chunk size and overlap work together. If the chunk size is 500 tokens and the overlap is 50 tokens, the next chunk starts 450 tokens after the previous start.

Increasing overlap can protect information near boundaries, but it also increases repeated content, storage, embedding work, and potentially retrieval redundancy.

Python: Compare Candidate Sizes

A simple experiment can show how many chunks different sizes create for the same document.

def chunk_words(words, size):
    return [words[i:i + size] for i in range(0, len(words), size)]

text = ("RAG retrieves relevant context before generation. "
        "Chunk size affects retrieval precision and context. "
        "Good evaluation uses realistic user questions.")
words = text.split()

for size in [5, 10, 15]:
    chunks = chunk_words(words, size)
    print(size, "words ->", len(chunks), "chunks")

This does not prove which size is best. It gives you candidates to test with embeddings and real retrieval questions.

Chunk Size and Retrieval Precision

Suppose a user asks, “What is the refund window for damaged products?” A tiny chunk containing only “30 days” may be hard to interpret. A huge chunk containing an entire returns policy may include many unrelated rules.

Chunk Size and Context Window

Your generator may support a large context window, but that does not mean every retrieved chunk should be large. Retrieval quality still matters.

If you retrieve several large chunks, their combined size can quickly consume the available context. Smaller, focused chunks can sometimes let the system retrieve more distinct evidence.

Chunk Size by Use Case

A Practical Starting Strategy

There is no universal number, but you can use a controlled process instead of guessing.

How to Evaluate Chunk Size

Build a small evaluation set with questions and expected evidence. Compare candidate chunk sizes using metrics such as retrieval hit rate, ranking position, answer correctness, and context usefulness.

results = {
    250: [1, 1, 0, 1, 1],
    500: [1, 1, 1, 1, 1],
    800: [1, 0, 1, 1, 0],
}

for size, hits in results.items():
    hit_rate = sum(hits) / len(hits)
    print(f"{size} tokens: {hit_rate:.0%} hit rate")

The example is intentionally simple. Real evaluation should use a larger, representative test set and inspect whether retrieved context actually supports the final answer.

Common Mistakes

• Using one size everywhere: different document types may need different strategies.
• Optimizing only for chunk count: fewer chunks does not automatically mean better retrieval.
• Ignoring structure: splitting headings, definitions, or question-answer pairs can damage meaning.
• Using excessive overlap: repeated content can increase cost and retrieval noise.
• Testing one query: a single successful retrieval is not enough evidence.
• Ignoring the generator: retrieved context must fit the overall context budget.

Chunk Size Is a System-Level Decision

Chunk size interacts with chunking strategy, overlap, embedding model, vector index, retrieval method, reranking, metadata filters, and the language model that generates the final answer.