ZerdaZerda

ContextAwareChunker

On this page

Module: chunks.context_aware_chunker
Import: from zerda.chunks import ContextAwareChunker

Purpose

ContextAwareChunker prevents mid-thought cuts by using a sliding window: each chunk contains not just its core sentences but also surrounding sentences for context. Candidate context sentences are no longer added blindly — each one is embedded and compared against the group's semantic centroid, and only kept if it's actually related to the group's meaning. This ensures downstream LLMs receive relevant context near chunk boundaries, without dragging in unrelated sentences just because they're nearby.

Use when:

  • Text is continuous narrative and context at boundaries matters
  • Downstream RAG requires overlap for coherence
  • You want window context that's semantically relevant, not just positionally adjacent

Algorithm

code · 11 lines
1. Split text into sentences (regex, Arabic-aware).
2. Compute a sentence embedding for every sentence (encode once, reused below).
3. _group_sentences(): accumulate into primary groups up to chunk_size,
   carrying overlap sentences between groups.
4. _apply_window(): for each primary group,
   a. Compute the group's embedding (centroid = mean of its sentences' embeddings).
   b. Collect up to window_size candidate sentences before and after the group.
   c. _filter_by_similarity(): keep a candidate only if its cosine similarity
      to the group centroid exceeds similarity_threshold; drop it otherwise.
   d. Merge kept context sentences with the group's core sentences.
5. Emit each windowed group as a WINDOW ChunkType chunk.

Constructor

Python · 11 lines
ContextAwareChunker(
    config: Optional[ChunkConfig] = None,
    *,
    chunk_size: int = 512,
    overlap: int = 128,
    window_size: int = 2,
    min_chunk_size: int = 50,
    language: str = "auto",
    embedding_model: str = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    similarity_threshold: float = 0.75,
)
ParameterTypeDefaultDescription
configChunkConfig | NoneNoneReads chunk_size, overlap, context_window_size, min_chunk_size from config
chunk_sizeint512Primary group size limit
overlapint128Overlap in characters between primary groups
window_sizeint2Max extra sentences considered before/after each group (before semantic filtering)
min_chunk_sizeint50Minimum chunk character length
languagestr"auto"Stored in metadata
embedding_modelstr"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"Multilingual sentence-transformers model used to embed sentences and groups
similarity_thresholdfloat0.75Minimum cosine similarity a candidate context sentence must have with the group centroid to be included

All chunks have ChunkType.WINDOW.

Dependency: requires sentence-transformers and numpy. The embedding model is loaded once per process (cached) and reused across ContextAwareChunker instances.

Note: because each candidate is filtered individually rather than as a contiguous run, the kept context sentences around a group are not guaranteed to be contiguous with it — a nearby sentence can be dropped while a slightly farther one (that's more on-topic) is kept.

Examples

Python · 4 lines
from zerda.chunks import ContextAwareChunker

chunker = ContextAwareChunker(window_size=3, chunk_size=400)
chunks = chunker.chunk(narrative_text, source="novel.txt")
Python · 3 lines
# Stricter semantic filtering: only very closely related context sentences are kept
chunker = ContextAwareChunker(similarity_threshold=0.85)
chunks = chunker.chunk(narrative_text, source="novel.txt")

Related pages