What is Chunking?

What Is Chunking?

Chunking means dividing a larger piece of content into smaller sections called chunks. A chunk can be a paragraph, a group of sentences, a section, or another logical unit of information.

In a typical RAG system, documents are too large to treat as one retrieval unit. Instead, the system splits them into chunks, creates an embedding for each chunk, stores those embeddings, and later retrieves the chunks that are most relevant to a user's question.

Why Do We Need Chunking?

Suppose you have a 100-page product manual. A user asks, “How long is the warranty?” Sending the entire manual into every retrieval step is inefficient and can introduce unrelated information.

Where Chunking Fits in RAG

Chunking normally happens during the indexing or ingestion stage, before a user asks a question. The chunks are then embedded and stored for later retrieval.

The important connection is that retrieval happens over the chunks, not over the original document as one giant unit.

Document vs Chunk

A document is the original source. A chunk is a smaller retrievable piece derived from that source.

Chunk Size

Chunk size describes how much content each chunk contains. It can be measured using characters, words, tokens, sentences, or another unit depending on the chunking method.

Chunk Boundaries

Where a chunk starts and ends can be just as important as its size. Ideally, a chunk should avoid cutting an important idea in the middle.

Logical boundaries can come from headings, paragraphs, sections, lists, sentences, or document structure. Later lessons will cover specific chunking strategies.

Simple Python Chunking

Here is a basic example that splits text into fixed-size character chunks. It is intentionally simple so you can understand the core idea before learning more advanced strategies.

text = "Generative AI can answer questions from documents. " \
       "Chunking divides long documents into smaller pieces. " \
       "Those pieces can then be embedded and retrieved."

chunk_size = 60
chunks = [
    text[i:i + chunk_size]
    for i in range(0, len(text), chunk_size)
]

for number, chunk in enumerate(chunks, start=1):
    print(f"Chunk {number}: {chunk}")

Chunking With Simple Sentences

A sentence-aware approach can preserve meaning better than cutting text at an arbitrary character position. This example groups sentences until a target size is reached.

sentences = [
    "Chunking splits documents into smaller pieces.",
    "Each piece can be embedded separately.",
    "Retrieval can then return only relevant pieces.",
    "Good boundaries help preserve meaning."
]

max_sentences = 2
chunks = []

for i in range(0, len(sentences), max_sentences):
    chunks.append(" ".join(sentences[i:i + max_sentences]))

for chunk in chunks:
    print(chunk)

Chunk Size vs Retrieval Quality

Chunking creates a trade-off. Smaller chunks can make matching more focused, while larger chunks can preserve more context.

Chunking and Embeddings

After chunking, each chunk can be passed to an embedding model. The resulting vectors represent the chunks in a form that supports similarity search.

If a user's question is semantically related to Chunk A, similarity search can rank that chunk highly. Poor chunking can make the embedding less useful because the vector represents a mixture of too many unrelated ideas.

Chunking and RAG

In RAG, the retrieved chunks become context for the language model. This makes chunk quality important at two different points: retrieval and generation.

Chunking Is Not the Same as Summarization

Chunking and summarization solve different problems.

A RAG pipeline can use both, but they should not be treated as the same operation.

Common Chunking Mistakes

How Do You Know If Chunking Is Good?

Good chunking should be evaluated with real questions. Ask whether the retrieved chunks contain the information needed to answer each question.