ZerdaZerda

SemanticChunker

On this page

Module: chunks.semantic_chunker
Import: from zerda.chunks import SemanticChunker

Table of Contents

Purpose

SemanticChunker splits text by detecting natural semantic topic boundaries using sentence embeddings and cosine similarity. Rather than splitting at fixed character counts, it opens a new chunk when adjacent sentences are topically dissimilar (similarity drops below similarity_threshold).

Use when:

  • Text has clearly distinct topic sections
  • You want coherent, topic-complete chunks
  • Retrieval quality matters more than uniform chunk size

Avoid when:

  • No GPU or sentence-transformers package available (falls back to TF-IDF)
  • Text is homogeneous prose without topic shifts
  • Latency constraints disallow embedding calls

Algorithm

code · 9 lines
1. Split text into sentences (regex-based, no NLTK required).
   - Protects common abbreviations like "Dr." from false splits.
2. Compute per-sentence embeddings via EmbeddingProvider (batched, cached).
3. Compute cosine similarity between each consecutive sentence pair.
4. Open a new chunk when:
   - similarity[i] < similarity_threshold, OR
   - current chunk length >= max_chunk_size
5. Merge any resulting chunks shorter than min_chunk_size with the next chunk.
6. Wrap each chunk text in a DocumentChunk with SEMANTIC ChunkType.

Constructor

Python · 10 lines
SemanticChunker(
    config: Optional[ChunkConfig] = None,
    *,
    similarity_threshold: float = 0.75,
    min_chunk_size: int = 50,
    max_chunk_size: int = 2048,
    model_name: Optional[str] = None,
    device: Optional[str] = None,
    language: str = "auto",
)

Parameters

ParameterTypeDefaultDescription
configChunkConfig | NoneNoneIf provided, similarity_threshold, min_chunk_size, max_chunk_size, model_name, embedding_batch_size, and embedding_cache_size are read from config.
similarity_thresholdfloat0.75Cosine similarity below this triggers a new chunk. Higher = more chunks.
min_chunk_sizeint50Chunks shorter than this are merged with the next chunk.
max_chunk_sizeint2048Hard cap; chunk is cut even if similarity is above threshold.
model_namestr | NoneNoneSentence-transformers model. Defaults to config.semantic_model.
devicestr | NoneNoneTorch device ("cpu", "cuda", "mps").
languagestr"auto"Stored in chunk metadata.

Public Methods

Inherited from BaseChunker:

chunk(text, doc_id="", source="") → List[DocumentChunk]

Public entry point. Calls _chunk_impl() then _finalize().

chunk_documents(documents) → List[DocumentChunk]

Chunks a list of Document objects, preserving doc_id and source.

Internal Workflow

Complexity

DimensionValue
Time (sentence splitting)O(n) where n = text length
Time (embeddings)O(s × b) where s = sentence count, b = batch size
Time (similarity)O(s)
Time (merge)O(s)
SpaceO(s × d) where d = embedding dimension

Embedding computation dominates. Caching in EmbeddingProvider (LRU, default 10,000 entries) eliminates repeated computation for identical sentences.

Thread Safety

SemanticChunker itself holds no mutable shared state after construction. EmbeddingProvider is a per-model singleton with a plain-dict cache — not thread-safe for concurrent writes. Use one SemanticChunker instance per thread, or protect with a lock, when processing documents in parallel.

Examples

Basic

Python · 4 lines
from zerda.chunks import SemanticChunker

chunker = SemanticChunker()
chunks = chunker.chunk("Topic A sentences. Topic B sentences.", source="test.txt")

With Configuration

Python · 10 lines
from zerda.chunks import SemanticChunker, ChunkConfig

config = ChunkConfig(
    semantic_similarity_threshold=0.65,
    min_chunk_size=100,
    max_chunk_size=1024,
    semantic_model="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
)
chunker = SemanticChunker(config=config)
chunks = chunker.chunk(text, doc_id="paper_1", source="paper.pdf")

GPU Acceleration

Python · 2 lines
chunker = SemanticChunker(device="cuda")
chunks = chunker.chunk(long_text)

Adjusting Threshold

Python · 5 lines
# More chunks, tighter semantic cohesion
chunker_tight = SemanticChunker(similarity_threshold=0.85)

# Fewer, broader chunks
chunker_loose = SemanticChunker(similarity_threshold=0.55)

Error Handling (EmbeddingProvider Fallback)

Python · 4 lines
# No sentence-transformers installed → TF-IDF fallback, no exception
chunker = SemanticChunker()
chunks = chunker.chunk(text)
# chunks are valid, but boundaries based on TF-IDF similarity

Processing Documents

Python · 5 lines
from zerda.chunks import SemanticChunker, Document

chunker = SemanticChunker()
docs = [Document(page_content=t, metadata={"source": f"doc_{i}.txt"}) for i, t in enumerate(texts)]
all_chunks = chunker.chunk_documents(docs)

Best Practices

  • Keep similarity_threshold between 0.6 and 0.85 for most corpora.
  • Use ChunkConfig with embedding_cache_size=50_000 for large document sets to maximize cache hits.
  • Run on GPU (device="cuda") for corpora with thousands of documents.
  • Pair with MetadataEnricher for keyword extraction on the resulting chunks.

Common Mistakes

MistakeFix
Setting similarity_threshold=1.0Every sentence becomes its own chunk. Use 0.65–0.85.
Not setting min_chunk_sizeVery short "leftover" sentences become isolated chunks. Set min_chunk_size=50.
Using for homogeneous textSimilarity stays high; all text ends up in one big chunk. Use StructureAwareChunker instead.

Related pages