Fixed-size Chunking

What Is Fixed-size Chunking?

Fixed-size chunking divides text into pieces according to a predetermined size. The size might be measured in characters, words, or tokens.

For example, if you choose 500 characters, the chunker repeatedly takes up to 500 characters from the source text until the document is consumed. The method does not first try to understand whether a sentence or section has finished.

How It Works

The basic algorithm is straightforward: choose a size, start at the beginning of the text, take that many units, move forward, and repeat.

If the document has 2,400 characters and the chosen size is 600 characters, a simple non-overlapping implementation produces up to four chunks.

Character-based Chunking

Character-based chunking is the easiest version to understand. The chunk size is a number of characters.

text = "Chunking splits long documents into smaller pieces for retrieval."
chunk_size = 20

chunks = [
    text[i:i + chunk_size]
    for i in range(0, len(text), chunk_size)
]

for number, chunk in enumerate(chunks, start=1):
    print(f"Chunk {number}: {chunk}")

Character Size Does Not Mean Token Size

A character count and a token count are different measurements. Language models process text as tokens, while a character-based chunker counts characters.

Token-based Fixed-size Chunking

Instead of counting characters, a system can tokenize the text and create chunks containing a maximum number of tokens. This is useful when you need tighter control over model context limits.

Python: Fixed-size Words

Here is a simple word-based version. It is useful for learning the algorithm, but production systems should consider the tokenizer and document structure.

text = "Python makes fixed size chunking easy to understand."
words = text.split()
chunk_size = 4

chunks = [
    " ".join(words[i:i + chunk_size])
    for i in range(0, len(words), chunk_size)
]

for chunk in chunks:
    print(chunk)

This version groups four words at a time. Notice that it can still cut across a sentence or logical section.

The Problem With Hard Boundaries

A fixed window can end in the middle of a sentence, list item, code example, or explanation.

Fixed-size chunking is therefore simple rather than universally intelligent. It can be a strong baseline, but it should be tested against your actual documents and questions.

Why Use Overlap?

Overlap repeats some content between neighboring chunks. The goal is to reduce the chance that an important idea is split exactly at a chunk boundary.

Python: Fixed Size With Overlap

A simple character-based implementation can move forward by chunk_size - overlap instead of the full chunk size.

text = "Fixed size chunking can use overlap to preserve nearby context."
chunk_size = 30
overlap = 8
step = chunk_size - overlap

chunks = [
    text[i:i + chunk_size]
    for i in range(0, len(text), step)
]

for number, chunk in enumerate(chunks, start=1):
    print(f"Chunk {number}: {chunk}")

Choosing a Chunk Size

There is no universally correct fixed size. A useful starting point depends on the document type, question style, embedding model, reranker, context window, and retrieval evaluation.

Fixed-size Chunking in RAG

In a RAG pipeline, fixed-size chunks are usually created during ingestion. Each chunk can then be embedded, stored, and retrieved later.

If the warranty sentence is split between two chunks, retrieval may need to return both pieces for the model to see the complete answer. Overlap can reduce this boundary problem.

Metadata for Fixed-size Chunks

Chunking should not discard the identity of the original source. Store useful metadata with every chunk so retrieval results can be traced back to the document.

When Fixed-size Chunking Works Well

It becomes less attractive when document structure carries important meaning, such as headings, tables, procedures, or tightly connected sections.

Common Mistakes

How to Evaluate Fixed-size Chunking

Compare several configurations instead of guessing. For example, test multiple chunk sizes with and without overlap against a representative question set.

Fixed-size vs Other Chunking Methods

Fixed-size chunking is valuable because it is easy to understand, implement, benchmark, and improve. Later lessons will explore alternatives that make boundary decisions more context-aware.