Cache
On this page
An intelligent, multi-tenant, cost-aware semantic caching module for LLM applications.
Cache (module in zerda package: cache) sits in front of an LLM call and answers the
question "have we already paid to generate an answer like this one?" It combines four
independent caching techniques into a single pipeline:
- Exact-match caching — an in-process LRU (
L1) plus a durable backend (SQLite, Redis, or in-memory) for identical, normalized queries. - Semantic caching — a vector similarity index (FAISS or a NumPy fallback) that matches queries which are worded differently but mean the same thing.
- Reinforcement-learning-based ranking and eviction — a Thompson Sampling multi-armed bandit that learns, from explicit feedback, which cached answers are actually good, and uses that signal both to rank semantic candidates and to decide what to evict.
- Cost-aware routing — a
DecisionEnginethat only serves a cache hit when doing so is a better trade-off (cost, quality, latency) than calling the LLM again.
Alongside this, the module ships a second, independent, simpler component —
MultiLevelCache — a generic three-tier (memory / memory / disk)
cache with pluggable eviction strategies (LRU, LFU, FIFO, TTL, Adaptive). It has no knowledge
of LLMs, embeddings, or tenants, and can be used on its own as a general-purpose cache.
Two modules in one idea
Both are exported from the top-level cache package and can be used independently or side
by side.
Where to go next
- New to the module? Start with Installation and the Quickstart.
- Want the big picture first? Read Architecture.
- Configuring for production? See Configuration.
- Looking for a specific class or function? Go to the API Reference.
- Want a guided, step-by-step walkthrough? See Tutorial: Getting Started, Tutorial: Multi-Tenant Setup, and Tutorial: Custom Storage Backend.
- Have a specific question? Check the FAQ.
- Curious about rough edges found during analysis? See Known Issues & Caveats.