LLM Routing
Filter, Score, Select, and ExplainLLM Routing
Filter, Score, Explain
Filter eligible routes; score quality, cost, and latency; select and explain.
Published: March 21, 2026 · By the QVeris Team
TL;DR
Modality, context, parameters, region, privacy, policy, capacity and budget define which routes are allowed.
Quality, cost, TTFT, total latency, throughput, availability and cache value can rank only eligible routes.
Provider fallback reacts after a failure; proactive routing can select a different model or endpoint before the first attempt.
Freeze workload classification, policy version, eligible set, scores, selected route, attempts, usage, cost and outcome.
The LLM routing pipeline
The router classifies the workload, applies hard filters, retrieves candidate metadata, scores eligible routes, selects one, and executes through a provider adapter. Feedback may update health signals immediately, while quality-policy changes should pass review and versioning.
Keep hard eligibility separate from optimization. Region, data policy, model lifecycle, required modality, context, structured output, tool support, latency ceiling, tenant entitlement, and maximum cost belong in a deterministic fail-closed gate. Only then should a scorer optimize quality, latency, price, health, or capacity. A high score must never rescue an ineligible endpoint.
Routing can occur at model level, provider level, or both. Model routing changes capabilities and output behavior. Provider routing selects among hosts for the same logical model, but host-specific snapshots, safety policy, regional processing, or throughput can still differ. Workload routing maps task classes to policies before either decision. Store all three identities—workload policy, logical model, and native provider endpoint—in the trace.
Common LLM routing strategies
| Strategy | Best fit | Verify before choosing |
|---|---|---|
| Provider fallback | Improve availability by trying an allowed alternate host after eligible failures. | Bound attempts, total deadline, provider equivalence, stream handling, charges and duplicate risk. |
| Model selection | Choose among models according to workload capability and product policy. | Model substitutions change quality, context, tools, safety, latency, price and user behavior. |
| Cost-aware routing | Minimize predicted cost while meeting hard quality and policy thresholds. | Use accepted-result cost and versioned prices; low token cost can increase failures and review. |
| Latency-aware routing | Prefer routes with acceptable TTFT and total latency for the workload and region. | Separate health from sparse noise; use windows, hysteresis and capacity signals to avoid thrashing. |
| Quality-aware routing | Predict which eligible model best satisfies task-specific quality requirements. | Evaluators need calibration, drift monitoring, abstention, offline proof and deterministic overrides. |
| Sticky cohort routing | Keep users or experiments on a stable route for consistency and measurement. | Define identity, expiry, failover, policy override, privacy and experiment-analysis rules. |
A governed router has four layers
Versioned model and endpoint capabilities, policy, price, region, health, lifecycle and provenance.
Deterministic filters that prevent optimization from selecting an illegal, unsupported or unsafe route.
Explicit objectives, weights, thresholds, normalization, tie-breaking, hysteresis and fallback.
Decision log, provider attempts, usage, cost, latency, quality, user result and reviewed policy promotion.
Build a routing benchmark
A routing benchmark must compare the router against simple baselines, not merely compare candidate models. Include a fixed primary route, a fixed low-cost route, and a deterministic rules policy. Measure whether dynamic routing improves accepted-task quality, tail latency, availability, or cost enough to justify classification errors, extra latency, policy complexity, and evaluator maintenance.
- Label workload classes, hard constraints, accepted-quality thresholds, latency targets, maximum cost, and abstain behavior.
- Run pinned baselines for every model and provider endpoint before dynamic routing, using the same prompts and acceptance rubric.
- Evaluate classification accuracy, eligible-set correctness, route regret, fallback success, retry amplification, accepted-output cost, and tail latency.
- Simulate outage, gradual degradation, price change, capacity limit, policy change, stale catalog data, and evaluator drift.
- Replay historical traffic offline, shadow decisions without affecting users, then canary by workload and tenant.
- Require the trace to reproduce every policy input, eligible set, exclusion, score, route, attempt, fallback, and final outcome.
Separate eligibility from optimization
Eligibility is a deterministic, fail-closed gate over versioned metadata. Scoring operates only on the frozen eligible set. Execution adapters cannot silently widen that set. Online health can remove an endpoint immediately; adding or materially changing a route requires reviewed catalog and policy updates. The trace stores every input used by the decision.
Production rule: routing is policy, not magic: the same frozen inputs and policy version should reproduce the same eligible set and decision.
LLM routing chooses reasoning; QVeris chooses action
An LLM router selects how the agent reasons. QVeris then Discovers, Inspects and Calls the external API, tool, service or live dataset that can execute the intended action. Join their evidence in one workflow trace, but keep policies and retries separate.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
It selects a model, hosting provider or endpoint for a request using constraints, policy and operational signals.
Routing chooses before or during execution; fallback is a bounded reaction after an eligible failure.
Yes, when cheaper eligible routes maintain accepted quality and savings exceed retries, review, platform and engineering cost.