Concepts

Handlers

A handler is the unit of business logic that ultimately answers a query. Every handler implements the BaseHandler abstract base class, which requires a single method:

def handle(self, request: HandlerRequest) -> HandlerResponse: ...

Optional overrides let a handler participate in async execution (handle_async), streaming (stream / stream_async), and pre-execution validation (validate).

Plain callables (Callable[[str], Any]) are also accepted anywhere a handler is expected — they're transparently wrapped in CallableHandler, which exists as a backward-compatibility shim for simpler function-based handlers.

Requests and responses

HandlerRequest and HandlerResponse are the typed envelope every handler receives and returns. Requests carry the raw query, arbitrary context, a correlation request_id, and router-internal metadata. Responses carry the payload, a success flag, an optional error message, handler-supplied metadata, and processing time.

HandlerResponse implements __bool__, so if response: is equivalent to if response.success:.

Routes

A Route is a single named destination: a description (used for semantic scoring), a set of example utterances, optional keyword signals, and a handler. Routes are not required to belong to a group — HierarchicalRouter.register() will auto-create a "default" group for ungrouped routes — but organizing routes into groups is the intended usage pattern.

Each Route tracks its own score_bias (updated by the feedback loop) and caches computed embeddings for its examples. Adding or removing an example invalidates the cached embeddings, requiring re-encoding before the next scoring pass picks them up.

Route groups

A RouteGroup is a named collection of routes sharing a common intent (e.g. "answer from documents" vs. "call a tool"). Groups have their own intent_examples and intent_keywords, scored the same way individual routes are, so the router can first narrow down to the right domain before comparing individual routes within it.

Three built-in templates cover common LLM application patterns:

• make_rag_group() — retrieval/document Q&A intent.
• make_tools_group() — tool/function-calling intent.
• make_chat_group() — general conversation (lowest priority by default — a catch-all).

Keywords

RouteKeywords attaches cheap, non-embedding signals to a route or group:

• required — all must appear in the query, or the route scores 0 (veto).
• excluded — if any appear, the route scores 0 (veto).
• any_of — each match adds boost to the score, normalized against the maximum possible bonus.
• boost — the per-keyword-match score contribution (default 0.05).

Per KeywordScorer.score: if any_of is empty and there are no required/excluded constraints either, the route gets a flat 0.1 base score (not vetoed, but no real signal).

Scoring

Three independent scorers implement BaseScorer:

• SemanticScorer — cosine (or dot-product, or Euclidean-derived) similarity between the query embedding and the maximum-similarity example embedding for each route.
• KeywordScorer — the keyword rules above; essentially free to compute.
• LLMScorer — an optional LLM classification call, invoked only as a fallback when semantic+keyword scores are low.

HybridScorer combines all three into one weighted score per ScoringConfig, then adds each route's persistent score_bias.

Confidence

A combined score is translated into a ConfidenceLevel — HIGH, MEDIUM, LOW, or NONE — by comparing against three configurable thresholds. Confidence determines the pipeline's next action: route directly, route with a note, fall back to the LLM scorer, or give up and hit the no-match path. See Architecture → Confidence tiers.

Routing results and traces

RoutingResult is what callers of route_query / route_async get back. It wraps a HandlerResponse plus a RoutingTrace — a full audit trail of the decision, including every scored candidate, timing breakdowns for embedding/scoring/execution, and free-text notes the pipeline attached along the way. This means a caller can always answer "why did it route here?" without extra instrumentation.

MultiRoutingResult wraps the results of parallel or multi-candidate execution.

Execution modes

ExecutionConfig.mode controls how the selected candidate(s) are actually run:

• SINGLE — execute only the top candidate, with retry.
• SEQUENTIAL — try candidates in ranked order, stopping at the first success.
• PARALLEL — run all candidates concurrently (via a thread pool for sync, asyncio.gather for async) and aggregate with a BaseAggregator (FirstWinsAggregator or MergeAggregator).

Feedback and adaptive bias

The FeedbackEngine tracks a per-route exponential moving average (EMA) of success/failure outcomes and derives a bounded score_bias ([-0.3, +0.3]) from it. This bias is added directly into the combined score during the next scoring pass, so routes that consistently succeed get a durable edge in close decisions, and routes that consistently fail get penalized — without retraining any model. Feedback can be recorded automatically (HierarchicalRouter._post_route, when FeedbackConfig.enabled) or manually via router.feedback(...).

Caching

Two independent caches exist:

• Route decision cache (CacheManager) — an LRU+TTL cache mapping a hash of the normalized query to (route_name, group_name). A hit skips embedding and scoring entirely.
• Embedding cache (inside EmbeddingProvider) — an unbounded in-memory dict mapping an MD5 hash of raw text to its embedding vector, avoiding repeated model calls for the same text.

Observability

RouterLogger emits structured (optionally JSON) log lines for routing decisions, cache hits, no-match events, slow routes, and feedback recordings. MetricsCollector maintains rolling-window latency, match-rate, cache-hit-rate, confidence-distribution, and per-route statistics, retrievable via router.metrics().