ZerdaZerda

Configuration Reference — ChunkConfig

On this page

ChunkConfig is a Python dataclass that centralises every tunable parameter in the system. Pass an instance to ChunkManager, or to any individual chunker, to customise behaviour.

Python · 7 lines
from zerda.chunks import ChunkConfig

config = ChunkConfig(
    chunk_size=512,
    overlap=128,
    use_semantic_chunking=True,
)

Table of Contents

Basic Sizing

ParameterTypeDefaultDescription
chunk_sizeint512Target chunk size in characters
overlapint128Overlap between adjacent chunks in characters
min_chunk_sizeint50Minimum chunk size; shorter chunks are dropped or merged
max_chunk_sizeint2048Maximum chunk size; larger sections are split further
strict_size_limitboolTrueIf True, force-splits chunks that exceed chunk_size × (1 + size_tolerance)
size_tolerancefloat0.05Fractional tolerance above chunk_size before strict_size_limit triggers (5%)

Tip: overlap must be less than chunk_size. Providing overlap >= chunk_size raises ValueError in chunkers that validate inputs.

Semantic Chunking

ParameterTypeDefaultDescription
use_semantic_chunkingboolFalseEnable embedding-based boundary detection
semantic_similarity_thresholdfloat0.75Cosine similarity below this value opens a new chunk
semantic_modelstr"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"Model for SemanticChunker
model_namestrsame as semantic_modelAlias kept for backward compatibility with ArabicTextChunker and MultilanguageTextChunker
embedding_batch_sizeint64Sentences per batch during embedding
embedding_cache_sizeint10_000LRU cache capacity (number of unique texts) in EmbeddingProvider

Adaptive Chunking

ParameterTypeDefaultDescription
use_adaptive_chunkingboolFalseEnable density-aware dynamic sizing
adaptive_technical_thresholdfloat0.6Information density score above which text is classified as "technical"
adaptive_min_sizeint100Minimum chunk size in adaptive mode
adaptive_max_sizeint1500Maximum chunk size in adaptive mode
adaptive_base_sizeint512Base (neutral density) chunk size

Smart Overlap

ParameterTypeDefaultDescription
use_smart_overlapboolTrueSelect overlap sentences by semantic similarity instead of raw character count
smart_overlap_thresholdfloat0.7Minimum cosine similarity for a sentence to be included in the overlap

Structure-Aware Chunking

ParameterTypeDefaultDescription
respect_document_structureboolTrueParse and respect Markdown/HTML structure
split_on_headersboolTrueOpen a new chunk at each Markdown/HTML heading
split_on_paragraphsboolTrueSplit at paragraph boundaries
preserve_code_blocksboolTrueKeep code blocks as single chunks regardless of size
preserve_tablesboolTrueKeep table sections as single chunks
preserve_listsboolTrueKeep list sections as single chunks

Context-Aware / Window Merging

ParameterTypeDefaultDescription
use_context_windowboolFalseEnable sliding-window context chunking
context_window_sizeint2Number of sentences to add before and after each primary group
min_context_sentencesint3Minimum sentences required before applying window logic

Metadata and Scoring

ParameterTypeDefaultDescription
extract_keywordsboolTrueRun TF-IDF keyword extraction on each chunk
keyword_max_countint10Maximum keywords stored per chunk
compute_chunk_scoresboolTrueCompute per-chunk importance score (0–1)
header_score_boostfloat1.5Score multiplier applied to header chunks

Deduplication

ParameterTypeDefaultDescription
deduplication_enabledboolTrueEnable deduplication stage in ChunkManager
dedup_similarity_thresholdfloat0.95Cosine similarity above which two chunks are considered duplicates
dedup_use_hashboolTrueUse fast SHA-256 hash matching before similarity check

Query-Aware Strategy

ParameterTypeDefaultDescription
query_awareboolFalseEnable automatic strategy selection based on sample queries
default_query_patternstr"general"Fallback strategy name: "faq", "documentation", "technical", "narrative", "general"

Performance

ParameterTypeDefaultDescription
async_processingboolFalseReserved for future async support
n_workersint4Reserved for future parallel processing

Note: async_processing and n_workers are declared in ChunkConfig but not yet wired to any async implementation in the current codebase.

Formatting and Text Cleanup

ParameterTypeDefaultDescription
preserve_formattingboolTrueKeep whitespace structure during chunking
fix_spacingboolTrueFix missing spaces around punctuation (used by ArabicTextChunker)

Tiktoken

ParameterTypeDefaultDescription
model_tokenstr"cl100k_base"tiktoken encoding name used by TokenTextSplitter

Language-Specific BERT Models

Pre-configured Hugging Face model names for MultilanguageTextChunker and ArabicTextChunker:

ParameterDefault Model
arabicCAMeL-Lab/bert-base-arabic-camelbert-mix
multilingualxlm-roberta-base
englishbert-base-uncased
chinesebert-base-chinese
frenchcamembert-base
germanbert-base-german-cased
spanishdccuchile/bert-base-spanish-wwm-uncased
russianDeepPavlov/rubert-base-cased
japanesecl-tohoku/bert-base-japanese
koreanklue/bert-base
portugueseneuralmind/bert-base-portuguese-cased
italiandbmdz/bert-base-italian-cased
dutchGroNLP/bert-base-dutch-cased
polishdkleczek/bert-base-polish-cased
turkishdbmdz/bert-base-turkish-cased
vietnamesevinai/phobert-base
hindiai4bharat/indic-bert

Transformer Parameters

ParameterTypeDefaultDescription
max_lenint512Maximum token length passed to BERT tokenizers
paddingboolTruePad sequences to max_len
truncationboolTrueTruncate sequences exceeding max_len
return_tensorstr"pt"PyTorch tensor format for tokenizer output

Merge Parameters

ParameterTypeDefaultDescription
min_sentencesint30Minimum sentence count threshold (reserved for future merge logic)

Complete Default Reference

Python · 75 lines
from zerda.chunks import ChunkConfig

config = ChunkConfig(
    # Basic sizing
    chunk_size=512,
    overlap=128,
    min_chunk_size=50,
    max_chunk_size=2048,
    strict_size_limit=True,
    size_tolerance=0.05,

    # Semantic
    use_semantic_chunking=False,
    semantic_similarity_threshold=0.75,
    semantic_model="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    embedding_batch_size=64,
    embedding_cache_size=10_000,

    # Adaptive
    use_adaptive_chunking=False,
    adaptive_technical_threshold=0.6,
    adaptive_min_size=100,
    adaptive_max_size=1500,
    adaptive_base_size=512,

    # Smart overlap
    use_smart_overlap=True,
    smart_overlap_threshold=0.7,

    # Structure
    respect_document_structure=True,
    split_on_headers=True,
    split_on_paragraphs=True,
    preserve_code_blocks=True,
    preserve_tables=True,
    preserve_lists=True,

    # Context window
    use_context_window=False,
    context_window_size=2,
    min_context_sentences=3,

    # Metadata
    extract_keywords=True,
    keyword_max_count=10,
    compute_chunk_scores=True,
    header_score_boost=1.5,

    # Deduplication
    deduplication_enabled=True,
    dedup_similarity_threshold=0.95,
    dedup_use_hash=True,

    # Query-aware
    query_aware=False,
    default_query_pattern="general",

    # Performance
    async_processing=False,
    n_workers=4,

    # Formatting
    preserve_formatting=True,
    fix_spacing=True,

    # Tiktoken
    model_token="cl100k_base",

    # Transformers
    max_len=512,
    padding=True,
    truncation=True,
    return_tensor="pt",
)

Related pages