Recursive Chunking

What Is Recursive Chunking?

Recursive chunking tries to split text at the largest useful boundary first. If the resulting piece is still too large, it moves to the next smaller separator and splits again.

A common sequence is paragraph → line → sentence → word → character. The exact separators depend on the document type and the chunking library.

Why Use a Hierarchy of Separators?

Consider a technical document containing several paragraphs. A fixed-size splitter may cut through a paragraph even when a clean paragraph boundary is available.

This often produces chunks that are easier to interpret because the splitter tries to preserve natural document structure before making finer cuts.

How Recursive Splitting Works

Separator Order Matters

The order of separators expresses your preference for what should stay together. Put larger structural boundaries first and smaller boundaries later.

separators = [
    "\\n\\n",  # paragraph
    "\\n",    # line
    ". ",     # sentence
    " "       # word
]

for separator in separators:
    print(repr(separator))

With this order, the algorithm prefers paragraph boundaries. It only considers smaller boundaries when a larger piece cannot fit the target size.

Simple Recursive Chunker in Python

The following learning implementation demonstrates the core idea without depending on an external framework.

def recursive_split(text, max_words, separators):
    if len(text.split()) <= max_words:
        return [text.strip()]

    separator = separators[0]
    parts = [part.strip() for part in text.split(separator) if part.strip()]

    if len(parts) == 1 and len(separators) > 1:
        return recursive_split(text, max_words, separators[1:])

    chunks = []
    current = []
    current_words = 0

    for part in parts:
        part_words = len(part.split())

        if current and current_words + part_words > max_words:
            chunks.append(" ".join(current))
            current = []
            current_words = 0

        if part_words > max_words and len(separators) > 1:
            chunks.extend(recursive_split(part, max_words, separators[1:]))
        else:
            current.append(part)
            current_words += part_words

    if current:
        chunks.append(" ".join(current))

    return chunks

text = "Paragraph one has useful context. It contains several sentences.\\n\\nParagraph two explains the next idea. It also contains supporting detail."

chunks = recursive_split(text, max_words=12, separators=["\\n\\n", "\\n", ". ", " "])

for i, chunk in enumerate(chunks, start=1):
    print(f"Chunk {i}: {chunk}")

Chunk Size Still Matters

Recursive chunking does not remove the need for a target size. You still need a maximum word or token budget so retrieval remains focused and the LLM receives manageable context.

Recursive Chunking and Overlap

Overlap can repeat part of the previous chunk so information near a boundary remains available in the next chunk.

Python: Adding Simple Overlap

A simple post-processing approach can carry the final part of one chunk into the next. Production implementations should account for token budgets and avoid creating tiny duplicate fragments.

def add_word_overlap(chunks, overlap_words):
    result = []

    for index, chunk in enumerate(chunks):
        if index == 0:
            result.append(chunk)
            continue

        previous_words = chunks[index - 1].split()
        prefix = " ".join(previous_words[-overlap_words:])
        result.append((prefix + " " + chunk).strip())

    return result

chunks = [
    "RAG retrieves relevant documents for a query.",
    "The retriever sends useful context to the language model.",
    "The model uses that context to generate an answer."
]

for chunk in add_word_overlap(chunks, overlap_words=4):
    print(chunk)

Recursive vs Fixed-size Chunking

Recursive chunking adds structure awareness without requiring a full semantic model. That makes it a strong general-purpose baseline for many RAG applications.

Recursive vs Sentence Chunking

Recursive Chunking in RAG

The retrieved chunks become context for the generation step. Better chunk boundaries can improve the chance that the retrieved text contains a complete, useful explanation.

Preserve Metadata

Keep metadata with every recursive chunk so retrieval results can be traced back to the original document and section.

Where Recursive Chunking Works Well

Where Recursive Chunking Can Struggle

Common Mistakes

How to Evaluate Recursive Chunking

Use a small evaluation set of realistic questions and compare recursive chunking against your current strategy. The goal is not the prettiest chunks; it is better retrieval and better final answers.