Concepts

The chunk abstraction

The library never defines its own chunk class for indexing — it accepts any object and adapts to it via duck typing, wrapped in ScoredChunk. Specifically, ScoredChunk looks for these attributes on the wrapped object (self.chunk), in this order of precedence:

This means LangChain-style Document objects (which use .page_content) work out of the box alongside custom dataclasses that use .text.

content_hash and deduplication

Every ScoredChunk computes a SHA-256 hash of its whitespace-normalized, lowercased text in __post_init__. This hash is used for:

• Merging results from multiple retrievers in HybridRetriever (same chunk found by vector and keyword search gets merged into one ScoredChunk rather than appearing twice)
• Exact-duplicate removal in ChunkFilter._dedup_hash

Query intent and complexity

RuleBasedQueryAnalyzer classifies every query into one QueryIntent and one QueryComplexity, using regex and keyword-list matching only — there is no ML model involved.

• Intent (QueryIntent): FACTUAL, REASONING, COMPARATIVE, PROCEDURAL, CODE, SUMMARIZE, UNKNOWN. Determined by counting how many Arabic/English signal phrases for each intent appear in the normalized query text; the intent with the most matches wins, with FACTUAL as the tie-break/default when nothing matches.
• Complexity (QueryComplexity): LOW, MEDIUM, HIGH. Determined by a mix of word count and regex signals — see api/query_analyzer for the exact thresholds.

Complexity directly determines the context budget via QueryAnalyzerConfig.budget_low/medium/high (defaults: 800 / 2000 / 4000 characters).

Context budget

The "budget" is a length limit — by default measured in characters, not tokens — for the final composed context string. It flows through the pipeline like this:

1. RuleBasedQueryAnalyzer assigns an initial budget based on query complexity.
2. AdaptiveLearningSystem.suggest_budget() can shrink it if historical usage has been consistently far below the assigned budget (see api/guard_and_learning).
3. SmartContextComposer packs chunks greedily until the budget would be exceeded, then stops (truncating the chunk in progress if no chunk fit yet at all).

You can switch the unit from characters to LLM tokens by passing a length_counter to ContextEngine, built via make_token_counter:

from zerda.context import ContextEngine, make_token_counter
import tiktoken

tokenizer = tiktoken.get_encoding("cl100k_base")
engine = ContextEngine(length_counter=make_token_counter(tokenizer))

make_token_counter expects a tokenizer object with an .encode(text) method — this matches tiktoken's interface exactly.

Source quality

SourceQuality is a 4-value enum (HIGH=3, MEDIUM=2, LOW=1, UNKNOWN=0) that you assign per source via ContextEngine.index(..., source_quality_map={"my-source": SourceQuality.HIGH}). It feeds into CompositeRanker's composite score (normalized to [0,1] by dividing by 3). If you never provide a map, every chunk defaults to SourceQuality.UNKNOWN (0), contributing nothing to the ranking score regardless of its configured weight.

Reciprocal Rank Fusion (RRF)

When more than one retrieval method returns a given chunk, HybridRetriever adds an RRF bonus on top of the weighted composite score, so that chunks confirmed by multiple retrieval signals are boosted — even if no single method ranked them highest. The exact formula and constant (RRF_K = 60) are documented in api/retriever.

Language handling

The library has built-in heuristics for Arabic alongside English, used throughout query_analyzer.py:

• Arabic detection is a simple ratio: count Arabic-range Unicode characters (\u0600-\u06FF) versus total non-space characters, compared against QueryAnalyzerConfig.arabic_threshold (default 0.3).
• Normalization strips Arabic diacritics (tashkeel) and unifies different forms of alef (إأآا → ا).
• Stop-word lists, intent signal phrases, and question-type prefixes are maintained separately for Arabic and English.
• The composer's "arabic" template (the default) prints Arabic UI strings (📚 المعلومات المسترجعة:, ─── نهاية السياق ───); the "english" template is the equivalent in English; "minimal" and "structured" are language-neutral.

Pluggable components

Every pipeline stage is defined as an ABC in strategy. This is the contract that lets you replace any default with your own implementation (an LLM-based query analyzer, a Pinecone/FAISS-backed retriever, a learned reranker, etc.) without changing ContextEngine itself. See Architecture → Component substitution points.

Legacy compatibility layer

legacy_compat.py and the legacy methods on ContextEngine (build(), build_multi_query_context(), compress_context(), get_context_stats()) exist to support an older, tuple-based API (List[Tuple[chunk, score]] instead of List[ScoredChunk]). ContextManager is a subclass of ContextEngine kept for the same reason — see api/legacy_compat for details, including a documented quirk in __init__ around these symbols.