ZerdaZerda

AdaptiveChunker

On this page

Module: chunks.adaptive_chunker
Import: from zerda.chunks import AdaptiveChunker

Table of Contents

Purpose

AdaptiveChunker adjusts chunk size dynamically based on the estimated information density of the text. Dense technical text (code, formulas, SQL) gets smaller chunks for precision retrieval; narrative/simple text gets larger chunks for context continuity.

Use when:

  • A document mixes technical sections with prose (e.g., a research paper with algorithm descriptions and introductions)
  • You want automatically appropriate chunk sizes without manually tuning per-document-type

Algorithm

code · 8 lines
1. Estimate information density of the full text (0–1 float).
2. Compute target_size via linear interpolation:
   target_size = max_size - density × (max_size - min_size)
3. Split text into sentences (regex: .!?؟ or double newline).
4. Accumulate sentences into a chunk until local_target would be exceeded.
5. On overflow: emit chunk, carry over last N chars as overlap.
6. Flush remaining sentences as the final chunk.
7. Each chunk carries density value in metadata.extra["density"].

Density → Target Size Interpolation

DensityTarget Size (defaults)
0.0 (pure narrative)1500 (max)
0.5800
1.0 (max technical)100 (min)

Information Density Estimation

The private _estimate_density(text) function returns a float in [0, 1]:

code · 6 lines
density = (
    min(tech_hits / (total_chars / 100), 1.0) × 0.4
  + unique_vocab_ratio                          × 0.3
  + avg_word_length_score                       × 0.2
  + arabic_complex_hits / (total_chars / 200)   × 0.1
)

Technical indicators (regex): programming keywords (def, class, SELECT, etc.), API/SDK/HTTP keywords, math terms (algorithm, theorem, O(n)), code punctuation ({}<>[]()=), decimal numbers.

Arabic complex indicator: Arabic words ≥ 7 characters (likely formal/technical register).

Constructor

Python · 10 lines
AdaptiveChunker(
    config: Optional[ChunkConfig] = None,
    *,
    base_size: int = 512,
    min_size: int = 100,
    max_size: int = 1500,
    overlap: int = 128,
    technical_threshold: float = 0.5,
    language: str = "auto",
)

Parameters

ParameterTypeDefaultDescription
configChunkConfig | NoneNoneIf provided, reads adaptive_base_size, adaptive_min_size, adaptive_max_size, overlap, adaptive_technical_threshold from config.
base_sizeint512Target size at neutral density
min_sizeint100Minimum chunk size (highest density)
max_sizeint1500Maximum chunk size (lowest density)
overlapint128Overlap in characters
technical_thresholdfloat0.5Stored but not used directly in splitting logic (density is continuous)
languagestr"auto"Stored in chunk metadata

Note: When config is passed, base_size, min_size, max_size, overlap, and technical_threshold constructor arguments are ignored and config values take precedence.

Public Methods

Inherited from BaseChunker:

chunk(text, doc_id="", source="") → List[DocumentChunk]

chunk_documents(documents) → List[DocumentChunk]

Metadata

Each chunk's metadata.extra contains:

Python
{"density": 0.3417}   # estimated density of the full document

metadata.chunk_type is always ChunkType.ADAPTIVE.

Complexity

DimensionValue
TimeO(n) — linear scan of sentences
SpaceO(k) — k = sentences in current buffer

No ML models required. Fully rule-based with regex.

Thread Safety

Stateless after construction. Thread-safe for concurrent calls with independent text inputs.

Examples

Basic

Python · 7 lines
from zerda.chunks import AdaptiveChunker

chunker = AdaptiveChunker()
chunks = chunker.chunk(mixed_document, source="report.pdf")

for c in chunks:
    print(f"density={c.metadata.extra['density']:.2f}  chars={c.char_count}")

Technical Document

Python · 10 lines
from zerda.chunks import AdaptiveChunker, ChunkConfig

config = ChunkConfig(
    adaptive_min_size=80,
    adaptive_max_size=800,
    adaptive_base_size=300,
    overlap=64,
)
chunker = AdaptiveChunker(config=config)
chunks = chunker.chunk(api_docs, source="api_docs.md")

Inspecting Density

Python · 7 lines
from zerda.chunks.adaptive_chunker import _estimate_density

code_text = "def fibonacci(n): return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)"
prose_text = "Once upon a time in a land far away there lived a young princess."

print(_estimate_density(code_text))   # ~0.7+ (technical)
print(_estimate_density(prose_text))  # ~0.1 (narrative)

Error Handling

Python · 3 lines
# Empty text → empty list, no exception
chunks = AdaptiveChunker().chunk("", source="empty.txt")
assert chunks == []

Best Practices

  • Effective for research papers, technical manuals, or any document that mixes prose introductions with code/equation sections.
  • Pair with MetadataEnricher to also get keyword scores.
  • Inspect metadata.extra["density"] to understand how the chunker classified your text.

Common Mistakes

MistakeFix
Setting min_size larger than max_sizeRaises no error but all chunks collapse to min_size. Ensure min_size < max_size.
Passing very short texts (< min_size)Short texts are dropped. Pad or lower min_chunk_size via config.

Related pages