Tutorial 04 — Semantic Chunking
On this page
Level: Intermediate
Requires: pip install sentence-transformers
What Is Semantic Chunking?
Traditional chunkers split at fixed character counts. Semantic chunking watches for topic drops — points where adjacent sentences become less similar — and uses those as chunk boundaries. The result: each chunk is topically coherent.
Basic Example
from zerda.chunks import SemanticChunker
chunker = SemanticChunker(similarity_threshold=0.75)
text = """
Neural networks are inspired by biological neurons. They consist of
layers of interconnected nodes that process information. Deep learning
uses many such layers to learn complex patterns.
The Amazon rainforest is the world's largest tropical forest. It covers
over 5.5 million square kilometers across nine countries. The rainforest
is home to an extraordinary diversity of plant and animal species.
"""
chunks = chunker.chunk(text, source="mixed_topics.txt")
print(f"Got {len(chunks)} chunks (expected: 2 distinct topics)")
for c in chunks:
print(f"\n--- Chunk {c.metadata.position} ---")
print(c.text[:100])
Tuning the Threshold
# Fine-grained: more chunks (higher semantic purity)
strict_chunker = SemanticChunker(similarity_threshold=0.85)
# Coarse-grained: fewer, broader chunks
loose_chunker = SemanticChunker(similarity_threshold=0.55)
strict_chunks = strict_chunker.chunk(text)
loose_chunks = loose_chunker.chunk(text)
print(f"Strict: {len(strict_chunks)} chunks")
print(f"Loose: {len(loose_chunks)} chunks")
Via ChunkManager
from zerda.chunks import ChunkManager, ChunkConfig, ChunkMode
config = ChunkConfig(
use_semantic_chunking=True,
semantic_similarity_threshold=0.72,
chunk_size=512,
min_chunk_size=100,
embedding_cache_size=50_000,
)
manager = ChunkManager(config=config, mode=ChunkMode.SEMANTIC)
chunks = manager.process(long_text, source="paper.pdf")
Embedding Cache
Repeated sentences across documents hit the cache and cost nothing:
from zerda.chunks import EmbeddingProvider
provider = EmbeddingProvider(cache_size=100_000)
chunker = SemanticChunker()
# Internal EmbeddingProvider re-used across calls in same session