ZerdaZerda

cache.metrics

On this page

Overview

Per the module docstring, this is the "production-grade metrics layer" for the semantic-cache pipeline, tracking: hit rates per layer (exact/semantic/tenant/LLM), latency histograms (p50/p90/p99), cost savings per tenant, RL reward distribution, eviction counts, per-tenant request counters, and error counters. The docstring states all operations are O(1) and thread-safe, and that histogram buckets use exponential binning for memory efficiency.

LatencyHistogram

Python · 2 lines
class LatencyHistogram:
    _N_BUCKETS = CacheConfig._N_BUCKETS   # 32

Overview

An approximate, fixed-memory latency histogram using exponential buckets: bucket i covers the range [2^i, 2^(i+1)) milliseconds, from [0, 1) up to [2^31, ...) (32 buckets, covering roughly up to 32-second latencies before the top bucket becomes a catch-all).

record(self, ms: float) -> None

Updates _count, _sum, _min, _max, and increments the appropriate bucket: idx = min(N_BUCKETS - 1, int(log2(max(1.0, ms)))). Thread-safe via RLock.

_percentile_locked(self, p: float) -> float (private)

Walks buckets in order, accumulating counts until the cumulative count reaches count * p / 100, and returns 2**i for the bucket where that threshold was crossed. This is an approximation — it reports the lower bound of the bucket a percentile falls into, not an interpolated exact value. Returns 0.0 if no samples have been recorded.

percentile(self, p: float) -> float

Public, locked wrapper around _percentile_locked.

snapshot(self) -> Dict[str, float]

Returns count, sum_ms, mean_ms, min_ms, max_ms, p50_ms, p90_ms, p99_ms (computed under a single lock acquisition for consistency across the whole snapshot).

Thread Safety

Every method acquires self._lock (RLock).

RollingCounter

Python · 2 lines
class RollingCounter:
    def __init__(self, window_seconds: int = 60) -> None

A thread-safe rolling sum over the last window_seconds seconds, backed by a deque of (timestamp, value) pairs.

record(self, value: float = 1.0) -> None

Appends (now, value) and evicts samples older than now - window_seconds from the front of the deque (the deque is time-ordered by construction, so this is a simple popleft() loop, not a full scan).

total(self) -> float

Sums the value of every sample within the current window. Note: this method independently filters by t >= cutoff rather than relying solely on record()'s eviction — so total() is correct even if record() hasn't been called recently enough to have swept out now-stale samples, though old samples that are never swept by a record() call will accumulate in memory until the next record() call runs its own cleanup.

Thread Safety

Both methods acquire self._lock (RLock).

CacheMetricsCollector

Python · 2 lines
class CacheMetricsCollector:
    def __init__(self) -> None

Overview

The central, single-instance-per-CacheManager metrics store (CacheManager._metrics). Holds hit/miss/eviction/error counters, five separate LatencyHistograms (overall, exact, semantic, llm, embedding), cost accumulators, a bounded reward-sample window (deque(maxlen=10_000)), and per-tenant RollingCounters / cost-saved totals.

Recording methods

MethodEffect
record_request(tenant_id="default")_total_requests += 1; records into the tenant's and the system-wide RollingCounter
record_exact_hit(latency_ms, tenant_id="default")_exact_hits += 1; records into latency_exact and latency_overall
record_exact_miss()_exact_misses += 1
record_semantic_hit(latency_ms, similarity, tenant_id="default")_semantic_hits += 1; records into latency_semantic/latency_overall (note: similarity is accepted as a parameter but not stored or aggregated anywhere in the analyzed source)
record_semantic_miss()_semantic_misses += 1
record_llm_fallback(latency_ms, cost_usd=0.0)_llm_fallbacks += 1; _total_spent_usd += cost_usd; records latency
record_embedding_latency(latency_ms)Records into latency_embedding only
record_cost_saved(usd, tenant_id="default")Adds to both the global _total_saved_usd and the per-tenant _tenant_cost_saved[tenant_id]
record_eviction(count=1)_evictions += count
record_expiration(count=1)_expirations += count
record_error(error_type: str)Increments _errors[error_type] (a defaultdict(int))
record_reward(reward: float)Appends to the bounded _reward_window deque

Every counter-mutating line is wrapped in with self._lock:; the histogram/rolling-counter record() calls happen outside that lock (each has its own internal lock), so there is a narrow window where, e.g., _exact_hits has been incremented but latency_exact hasn't yet recorded the corresponding sample, or vice versa, under concurrent access — not a correctness bug for the final snapshot values, but worth knowing if you're reading live, in-flight state from another thread.

snapshot(self) -> Dict[str, object]

Returns a nested dict of: top-level hit-rate percentages (overall_hit_rate, exact_hit_rate, semantic_hit_rate, tenant_hit_rate, llm_fallback_rate — all computed as count / max(1, total_requests), so they never divide by zero), raw counts, cost figures (including roi_multiplier = saved / max(0.000001, spent)), all five latency histogram snapshots, reward statistics (reward_mean, reward_p10, reward_p90 via the static _percentile_list helper), the rolling system-wide requests-per-second (rps_last_60s), and the raw errors dict.

[!NOTE] tenant_hit_rate and _tenant_hits are always zero As documented on RoutingDecision, _tenant_hits is declared and included in snapshot()'s overall_hit_rate numerator and tenant_hit_rate denominator math, but no code path in the analyzed source ever calls a method that increments it. Expect tenant_hit_rate to read 0.0 in practice.

tenant_snapshot(self, tenant_id: str) -> Dict[str, object]

Returns {tenant_id, rpm, cost_saved} for one tenant — a lighter-weight view than the full snapshot().

_percentile_list(values: List[float], p: int) -> float (static)

Sorts values in place and returns the value at index max(0, int(len(values)*p/100) - 1) — a simple, exact (not approximate/bucketed, unlike LatencyHistogram) percentile over whatever reward samples are currently in the window.

Examples

Python · 10 lines
from zerda.cache.metrics import CacheMetricsCollector

metrics = CacheMetricsCollector()
metrics.record_request("acme")
metrics.record_exact_hit(latency_ms=3.2, tenant_id="acme")
metrics.record_cost_saved(0.002, tenant_id="acme")

snap = metrics.snapshot()
print(snap["exact_hit_rate"], snap["total_saved_usd"])
print(metrics.tenant_snapshot("acme"))

Thread Safety

Every counter increment is protected by self._lock; the histogram and rolling-counter objects protect themselves independently, as noted above.

  • CacheManager — the sole owner; calls nearly every recording method
  • CostModel — supplies the cost figures passed to record_cost_saved/record_llm_fallback

Related pages