Semantic Chunking

What Is Semantic Chunking?

Semantic chunking creates chunks by looking for changes in meaning. Instead of saying “every 500 characters is a chunk,” the algorithm asks whether neighboring sentences are still talking about the same idea.

Why Semantic Boundaries Matter

A document can contain sentences of very different lengths. A fixed-size splitter may put two unrelated topics in the same chunk simply because they happen to be close in the original text.

This can improve retrieval because a returned chunk is more likely to contain a coherent idea rather than unrelated neighboring material.

Semantic Chunking Uses Embeddings

To compare meaning, sentences can be converted into embeddings. Each sentence becomes a vector, and neighboring vectors can then be compared with a similarity measure such as cosine similarity.

How Topic Boundaries Are Detected

A simple approach compares each sentence with the next sentence. If similarity stays high, the sentences can remain in the same chunk. If similarity falls below a chosen threshold, the algorithm can start a new chunk.

Similarity Threshold

The threshold controls how much semantic change is required before creating a new chunk. A high threshold creates more boundaries; a low threshold allows more sentences to stay together.

Simple Semantic Chunking in Python

The following example uses Sentence Transformers to create sentence embeddings and cosine similarity to detect topic changes. It is intentionally simple so you can see the mechanism.

!pip install -q sentence-transformers

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

sentences = [
    "A vector database stores embeddings for fast similarity search.",
    "It can return the most relevant vectors for a query.",
    "Metadata filters can narrow the search to a specific product.",
    "Our support team answers customer emails from 9 AM to 6 PM.",
    "Support requests are tracked with a ticket number."
]

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(sentences)

for i in range(len(sentences) - 1):
    score = cosine_similarity(
        [embeddings[i]],
        [embeddings[i + 1]]
    )[0][0]
    print(f"{i} -> {i + 1}: {score:.3f}")

Building Chunks From the Scores

Once neighboring similarities are available, you can use a threshold to decide where a new chunk starts. A production implementation should also enforce minimum and maximum chunk sizes.

def semantic_chunks(sentences, embeddings, threshold=0.55):
    chunks = [[sentences[0]]]

    for i in range(1, len(sentences)):
        score = cosine_similarity(
            [embeddings[i - 1]],
            [embeddings[i]]
        )[0][0]

        if score < threshold:
            chunks.append([sentences[i]])
        else:
            chunks[-1].append(sentences[i])

    return [" ".join(chunk) for chunk in chunks]

chunks = semantic_chunks(sentences, embeddings, threshold=0.55)

for index, chunk in enumerate(chunks, start=1):
    print(f"Chunk {index}: {chunk}\n")

Minimum and Maximum Chunk Size

Pure semantic similarity can produce awkward chunks if it is allowed to split whenever a score changes. Practical systems usually combine semantic boundaries with size constraints.

Semantic Chunking vs Other Strategies

Semantic Chunking in RAG

Semantic chunking happens during ingestion, before chunks are embedded and stored for retrieval. Better boundaries can make the retrieved context more coherent.

Semantic Chunking and Overlap

Semantic boundaries already try to keep related ideas together, so overlap is not automatically required. However, a small overlap can still help when important information crosses a detected boundary.

Metadata Still Matters

Semantic chunking decides where to split. Metadata tells your application where the chunk came from and how it was created.

Where Semantic Chunking Works Well

When Semantic Chunking Can Struggle

Common Mistakes

How to Evaluate Semantic Chunking

Evaluate the complete retrieval pipeline rather than optimizing chunk boundaries in isolation.