ZerdaZerda

Architecture

On this page

The cache module contains two independent subsystems that share the top-level namespace but do not call into each other:

  1. The semantic cache pipeline, centered on CacheManager.
  2. The generic tiered cache, MultiLevelCache.

They happen to share two superficially similar names — both have an "L1/L2" concept and a CacheEntry-like value class — but CacheManager uses IntelligentCacheEntry (cache.models) while MultiLevelCache uses the simpler CacheEntry (cache.cache_entry). These are not interchangeable.

[!NOTE] File name vs. contents The storage factory function build_storage() lives in a file named factury_fun (not factory) — this is how it appears in the source tree.

Module dependency graph

MultiLevelCache only depends on CacheConfig for its size/TTL defaults — it does not touch embedding_index, security, tenant_manager, policy_learner, or the storage/ package at all. Its own "L3" persistence is a private pickle-file mechanism inside multi_level_cache, unrelated to cache.storage.

CacheManager request pipeline

get() — lookup flow

Every branch records metrics via CacheMetricsCollector (hit/miss counters, latency histograms, cost saved) before returning a CacheLookupResult.

put() — store flow

Background eviction

A threading.Timer, restarted after every run (_start_eviction_timer → _run_eviction → _evict_cycle → _start_eviction_timer), fires every config.eviction.check_interval_s seconds (default 300s) and:

  1. Removes expired entries and entries older than eviction.max_age_s (if set).
  2. If total storage size exceeds a hard-coded 1 GB soft cap, computes a rough overflow-based eviction count; otherwise evicts the bottom 5% of entries by score.
  3. Asks CachePolicyLearner.eviction_ranking for the lowest-scoring entries (lowest = most expendable) and removes them from storage, L1, and the vector index, releasing tenant memory/entry quota.
  4. Applies CachePolicyLearner.auto_decay to all remaining entries, shrinking bandit arms toward their priors.

MultiLevelCache tier flow

set() always writes to L1 first; entries move down the hierarchy only via eviction (_evict_from_l1 → demote to L2 → _evict_from_l2 → demote to L3/disk → _evict_from_l3 → deleted permanently). Victim selection within a level is controlled by the configured CacheStrategy (LRU, LFU, FIFO, TTL, or ADAPTIVE).

Cross-cutting concerns

ConcernOwnerNotes
Thread safetythreading.RLock per component (MultiLevelCache.lock, _L1ExactCache._lock, EmbeddingIndex._lock, CacheMetricsCollector._lock, TenantManager._lock, CachePolicyLearner._lock, storage backends)Every stateful class documented here uses a re-entrant lock; none use multiprocessing-safe primitives (see Known Issues)
Async concurrencyasyncio.Lock in CachedEmbedder, threading.Event-based _InflightCoalescer in CacheManagerThe coalescer deliberately uses threading primitives (not asyncio.Event) so it works safely if callers run their own event loops in separate threads
SecuritySecurityGuardRegex-based prompt-injection/cache-poisoning detection, PII scrubbing, HMAC content signing, per-entry integrity verification
Multi-tenancyTenantManagerNamespacing, quotas (memory/entries/RPM), shared-cache opt-in
Cost accountingCostModel, DecisionEngineCharacter-based token approximation (÷4), USD cost estimates, utility-function-based routing
Learning / adaptationCachePolicyLearner, DecisionEngine.update_thresholdThompson Sampling bandit per entry; adaptive similarity threshold nudged by feedback
ObservabilityCacheMetricsCollector (pipeline), CacheMetrics (MultiLevelCache)Hit rates, latency percentiles (approximate exponential-bucket histogram), cost saved/spent, error counters

Related pages