Performance

On this page

Where time actually goes
Caching levers
Scoring cost scales with candidate count, not example count per route (for batch scoring)
LLM fallback is cost-gated, not latency-transparent
Execution mode tradeoffs
Retry cost
Metrics window size

Notes on performance characteristics observed directly in the source, to help you tune router for your workload.

Where time actually goes

Per RoutingTrace's timing breakdown, a routed request's latency splits into:

embedding_time_ms — one EmbeddingProvider.encode(query) call (cache-hit: near-zero; cache-miss: a full model forward pass).
scoring_time_ms — group selection (if >1 group) plus route scoring, both via vectorized NumPy operations in SemanticScorer.batch_score, plus a possible LLMScorer network call if triggered.
execution_time_ms — the handler's own work, plus any retries.

For most workloads, embedding computation dominates unless the query hits both the decision cache and the embedding cache.

Caching levers

Lever	Effect
`CacheConfig.enabled` (route decision cache)	A cache hit skips embedding and scoring entirely — the single biggest latency win for repeated queries.
`EmbeddingProvider`'s internal cache	Skips re-encoding the same text (query or example) — always on, unbounded, no TTL.
`CacheConfig.ttl_seconds` / `max_size`	Tune based on query repetition patterns and how quickly your feedback loop shifts `score_bias`.

See Architecture → Caching and CacheManager.

Scoring cost scales with candidate count, not example count per route (for batch scoring)

SemanticScorer.batch_score vectorizes similarity computation per route (stacking all of a route's example embeddings into one matrix operation), so adding more examples to a single route is cheap relative to adding more routes — more candidate routes means more per-route matrix operations in the batch loop. If you have many low-traffic routes, consider whether some can be merged or gated by enabled=False when not needed.

LLM fallback is cost-gated, not latency-transparent

HybridScorer only calls LLMScorer when the best semantic+keyword score is below fallback_confidence_threshold — this bounds LLM cost/latency to genuinely ambiguous queries. However, when it does trigger, it's a full network round-trip (openai.OpenAI().chat.completions.create(...)) scoring all candidates in a single prompt — for a large top_k or many candidate routes, this prompt (and its latency) grows with the candidate count.

Execution mode tradeoffs

Mode	Latency	Cost	Failure resilience
`SINGLE`	Lowest	Lowest	None — top candidate's failure is final.
`SEQUENTIAL`	Up to `k × (handler latency)` in the worst case (all candidates fail)	Proportional to attempts made	Falls through candidates until one succeeds.
`PARALLEL`	~`max(candidate latencies)`, bounded by `parallel_timeout`	Highest — all candidates always execute	Aggregates from whichever candidates succeed.

See ExecutionEngine for the full mode-by-mode behavior, including the sync/async asymmetries in timeout and retry handling.

Retry cost

ExecutionConfig.max_retries / retry_delay apply only on the synchronous execution path (_with_retry) — see the warning in ExecutionEngine. A misconfigured high max_retries combined with a long retry_delay can meaningfully inflate tail latency for routes with consistently failing handlers, since _with_retry blocks with time.sleep between attempts.

Metrics window size

ObservabilityConfig.metrics_window bounds the memory used by MetricsCollector's rolling deques and per-route latency lists — a larger window gives smoother rolling statistics at the cost of more memory (bounded, since it's a fixed-size deque per metric plus one list per route, each capped at metrics_window).

Related pages: Architecture, EmbeddingProvider, MetricsCollector