LLM Routing
Filter, Score, Select, and Explain什么是 LLM 路由？过滤、评分、选择并解释
LLM routing chooses a model or provider endpoint for each request. A governed router first removes ineligible candidates, then scores the remainder using versioned policy and records enough evidence to reproduce the decision.
LLM 路由为每个请求选择模型或供应商端点。受治理路由器先移除不合格候选，再用版本化策略为剩余候选评分，并记录足以重现决策的证据。
TL;DR
Modality, context, parameters, region, privacy, policy, capacity and budget define which routes are allowed.
Quality, cost, TTFT, total latency, throughput, availability and cache value can rank only eligible routes.
Provider fallback reacts after a failure; proactive routing can select a different model or endpoint before the first attempt.
Freeze workload classification, policy version, eligible set, scores, selected route, attempts, usage, cost and outcome.
模态、上下文、参数、区域、隐私、策略、容量与预算定义哪些路由被允许。
质量、成本、TTFT、总延迟、吞吐、可用性与缓存价值只能对合格路由排序。
供应商回退在失败后响应；主动路由可在首次尝试前选择不同模型或端点。
冻结负载分类、策略版本、合格集合、评分、所选路由、尝试、用量、成本与结果。
The LLM routing pipelineLLM 路由管线
The router classifies the workload, applies hard filters, retrieves candidate metadata, scores eligible routes, selects one, and executes through a provider adapter. Feedback may update health signals immediately, while quality-policy changes should pass review and versioning.
Router 对工作负载分类、应用硬性过滤、检索候选元数据、为合格路由评分、选择最终路由，再通过 Provider Adapter 执行。健康反馈可以立即更新，而质量策略变化应经过评审与版本化。
Keep hard eligibility separate from optimization. Region, data policy, model lifecycle, required modality, context, structured output, tool support, latency ceiling, tenant entitlement, and maximum cost belong in a deterministic fail-closed gate. Only then should a scorer optimize quality, latency, price, health, or capacity. A high score must never rescue an ineligible endpoint.
硬性资格判断必须与优化分离。区域、数据策略、模型生命周期、必需模态、上下文、结构化输出、工具支持、延迟上限、租户权限与最高成本都应进入确定性的 Fail-closed Gate。之后评分器才能优化质量、延迟、价格、健康或容量；再高的分数也不能让不合格端点重新进入候选。
Routing can occur at model level, provider level, or both. Model routing changes capabilities and output behavior. Provider routing selects among hosts for the same logical model, but host-specific snapshots, safety policy, regional processing, or throughput can still differ. Workload routing maps task classes to policies before either decision. Store all three identities—workload policy, logical model, and native provider endpoint—in the trace.
路由可以发生在模型级、Provider 级或两者。模型路由会改变能力与输出行为；Provider 路由为同一逻辑模型选择托管方，但托管方的 Snapshot、安全策略、区域处理或吞吐仍可能不同；工作负载路由则在两者之前把任务类别映射到策略。Trace 中应同时存储工作负载策略、逻辑模型与原生 Provider Endpoint 三种身份。
Common LLM routing strategies常见 LLM 路由策略
| Strategy策略 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Provider fallback供应商回退 | Improve availability by trying an allowed alternate host after eligible failures.在符合条件的失败后尝试获准备用托管方，提高可用性。 | Bound attempts, total deadline, provider equivalence, stream handling, charges and duplicate risk.限制尝试、总截止时间、供应商等价性、流处理、费用与重复风险。 |
| Model selection模型选择 | Choose among models according to workload capability and product policy.按负载能力与产品策略在模型间选择。 | Model substitutions change quality, context, tools, safety, latency, price and user behavior.模型替换会改变质量、上下文、工具、安全、延迟、价格与用户行为。 |
| Cost-aware routing成本感知路由 | Minimize predicted cost while meeting hard quality and policy thresholds.在满足质量与策略硬阈值时最小化预测成本。 | Use accepted-result cost and versioned prices; low token cost can increase failures and review.使用可接受结果成本与版本化价格；低 Token 成本可能增加失败与审核。 |
| Latency-aware routing延迟感知路由 | Prefer routes with acceptable TTFT and total latency for the workload and region.偏好满足负载与区域 TTFT 和总延迟要求的路由。 | Separate health from sparse noise; use windows, hysteresis and capacity signals to avoid thrashing.把健康与稀疏噪声分开，使用窗口、迟滞与容量信号避免抖动。 |
| Quality-aware routing质量感知路由 | Predict which eligible model best satisfies task-specific quality requirements.预测哪个合格模型最满足任务特定质量要求。 | Evaluators need calibration, drift monitoring, abstention, offline proof and deterministic overrides.评估器需要校准、漂移监控、弃权、离线验证与确定性覆盖。 |
| Sticky cohort routing粘性分组路由 | Keep users or experiments on a stable route for consistency and measurement.让用户或实验保持稳定路由，以维持一致性与测量。 | Define identity, expiry, failover, policy override, privacy and experiment-analysis rules.定义身份、过期、回退、策略覆盖、隐私与实验分析规则。 |
A governed router has four layers受治理路由器有四层
Versioned model and endpoint capabilities, policy, price, region, health, lifecycle and provenance.
Deterministic filters that prevent optimization from selecting an illegal, unsupported or unsafe route.
Explicit objectives, weights, thresholds, normalization, tie-breaking, hysteresis and fallback.
Decision log, provider attempts, usage, cost, latency, quality, user result and reviewed policy promotion.
版本化模型与端点能力、策略、价格、区域、健康、生命周期与来源。
确定性过滤，防止优化选择非法、不支持或不安全路由。
明确目标、权重、阈值、标准化、平局规则、迟滞与回退。
决策日志、供应商尝试、用量、成本、延迟、质量、用户结果与经评审策略发布。
Build a routing benchmark构建路由基准
A routing benchmark must compare the router against simple baselines, not merely compare candidate models. Include a fixed primary route, a fixed low-cost route, and a deterministic rules policy. Measure whether dynamic routing improves accepted-task quality, tail latency, availability, or cost enough to justify classification errors, extra latency, policy complexity, and evaluator maintenance.
路由基准不仅要比较候选模型，还要把 Router 与简单基线比较。至少加入固定主路由、固定低成本路由和确定性规则策略。应衡量动态路由对有效任务质量、尾延迟、可用性或成本的改善，是否足以抵消分类错误、额外延迟、策略复杂度与 Evaluator 维护成本。
- Label workload classes, hard constraints, accepted-quality thresholds, latency targets, maximum cost, and abstain behavior.
- Run pinned baselines for every model and provider endpoint before dynamic routing, using the same prompts and acceptance rubric.
- Evaluate classification accuracy, eligible-set correctness, route regret, fallback success, retry amplification, accepted-output cost, and tail latency.
- Simulate outage, gradual degradation, price change, capacity limit, policy change, stale catalog data, and evaluator drift.
- Replay historical traffic offline, shadow decisions without affecting users, then canary by workload and tenant.
- Require the trace to reproduce every policy input, eligible set, exclusion, score, route, attempt, fallback, and final outcome.
- 标注负载类别、硬性约束、有效质量阈值、延迟目标、最高成本与 Abstain 行为。
- 在动态路由前，为每个模型与 Provider Endpoint 运行固定基线，并使用相同 Prompt 与验收 Rubric。
- 评估分类准确率、合格集合正确性、Route Regret、Fallback 成功、重试放大、有效输出成本与尾延迟。
- 模拟中断、逐步劣化、价格变化、容量限制、策略变化、目录数据过期与 Evaluator Drift。
- 先离线回放历史流量，再以 Shadow 方式记录决策而不影响用户，最后按工作负载和租户灰度。
- 要求 Trace 可以重现每项策略输入、合格集合、排除、评分、路由、尝试、Fallback 与最终结果。
Separate eligibility from optimization分离资格判断与优化
Eligibility is a deterministic, fail-closed gate over versioned metadata. Scoring operates only on the frozen eligible set. Execution adapters cannot silently widen that set. Online health can remove an endpoint immediately; adding or materially changing a route requires reviewed catalog and policy updates. The trace stores every input used by the decision.
资格判断是基于版本化元数据的确定性 Fail-Closed 门禁；评分只在冻结的合格集合上运行；执行适配器不得静默扩大集合。在线健康可立即移除端点，但新增或重大改变路由需经目录与策略评审更新。调用链存储决策使用的所有输入。
Production rule: routing is policy, not magic: the same frozen inputs and policy version should reproduce the same eligible set and decision.
生产规则：路由是策略，不是魔法：相同冻结输入与策略版本应重现相同合格集合与决策。
LLM routing chooses reasoning; QVeris chooses actionLLM 路由选择推理，QVeris 选择行动
An LLM router selects how the agent reasons. QVeris then Discovers, Inspects and Calls the external API, tool, service or live dataset that can execute the intended action. Join their evidence in one workflow trace, but keep policies and retries separate.
LLM 路由器选择智能体如何推理；QVeris 随后发现、检查并调用能够执行预期行动的外部 API、工具、服务或实时数据集。两者证据连接在同一工作流调用链中，但策略与重试保持分离。
FAQ
It selects a model, hosting provider or endpoint for a request using constraints, policy and operational signals.
Routing chooses before or during execution; fallback is a bounded reaction after an eligible failure.
Yes, when cheaper eligible routes maintain accepted quality and savings exceed retries, review, platform and engineering cost.
使用约束、策略与运行信号，为请求选择模型、托管供应商或端点。
路由在执行前或执行中选择；回退是在符合条件失败后的有边界反应。
可以，前提是更便宜合格路由保持可接受质量，且节省超过重试、审核、平台与工程成本。