LLM Routing
Filter, Score, Select, and Explain
什么是 LLM 路由？过滤、评分、选择并解释
LLM routing chooses a model or provider endpoint for each request. A governed router first removes ineligible candidates, then scores the remainder using versioned policy and records enough evidence to reproduce the decision.
LLM 路由为每个请求选择模型或供应商端点。受治理路由器先移除不合格候选，再用版本化策略为剩余候选评分，并记录足以重现决策的证据。
TL;DR
Modality, context, parameters, region, privacy, policy, capacity and budget define which routes are allowed.
Quality, cost, TTFT, total latency, throughput, availability and cache value can rank only eligible routes.
Provider fallback reacts after a failure; proactive routing can select a different model or endpoint before the first attempt.
Freeze workload classification, policy version, eligible set, scores, selected route, attempts, usage, cost and outcome.
模态、上下文、参数、区域、隐私、策略、容量与预算定义哪些路由被允许。
质量、成本、TTFT、总延迟、吞吐、可用性与缓存价值只能对合格路由排序。
供应商回退在失败后响应；主动路由可在首次尝试前选择不同模型或端点。
冻结负载分类、策略版本、合格集合、评分、所选路由、尝试、用量、成本与结果。
The LLM routing pipeline LLM 路由管线
The router classifies the workload, applies hard filters, retrieves candidate metadata, scores eligible routes, selects one and executes through a provider adapter. Feedback may update health signals immediately, while quality-policy changes should pass review and versioning.
路由器分类负载、应用硬过滤、检索候选元数据、为合格路由评分、选择并通过供应商适配器执行。反馈可立即更新健康信号，而质量策略变化应经过评审与版本化。
Routing can occur at model level, provider level or both. Model routing changes capabilities and output behavior. Provider routing selects among hosts for the same logical model. Workload routing maps task classes to policies before either decision.
路由可发生在模型级、供应商级或两者。模型路由会改变能力与输出行为；供应商路由为同一逻辑模型选择托管方；工作负载路由在两者之前把任务类别映射到策略。
Common LLM routing strategies 常见 LLM 路由策略
| Strategy 策略 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Provider fallback 供应商回退 | Improve availability by trying an allowed alternate host after eligible failures. 在符合条件的失败后尝试获准备用托管方，提高可用性。 | Bound attempts, total deadline, provider equivalence, stream handling, charges and duplicate risk. 限制尝试、总截止时间、供应商等价性、流处理、费用与重复风险。 |
| Model selection 模型选择 | Choose among models according to workload capability and product policy. 按负载能力与产品策略在模型间选择。 | Model substitutions change quality, context, tools, safety, latency, price and user behavior. 模型替换会改变质量、上下文、工具、安全、延迟、价格与用户行为。 |
| Cost-aware routing 成本感知路由 | Minimize predicted cost while meeting hard quality and policy thresholds. 在满足质量与策略硬阈值时最小化预测成本。 | Use accepted-result cost and versioned prices; low token cost can increase failures and review. 使用可接受结果成本与版本化价格；低 Token 成本可能增加失败与审核。 |
| Latency-aware routing 延迟感知路由 | Prefer routes with acceptable TTFT and total latency for the workload and region. 偏好满足负载与区域 TTFT 和总延迟要求的路由。 | Separate health from sparse noise; use windows, hysteresis and capacity signals to avoid thrashing. 把健康与稀疏噪声分开，使用窗口、迟滞与容量信号避免抖动。 |
| Quality-aware routing 质量感知路由 | Predict which eligible model best satisfies task-specific quality requirements. 预测哪个合格模型最满足任务特定质量要求。 | Evaluators need calibration, drift monitoring, abstention, offline proof and deterministic overrides. 评估器需要校准、漂移监控、弃权、离线验证与确定性覆盖。 |
| Sticky cohort routing 粘性分组路由 | Keep users or experiments on a stable route for consistency and measurement. 让用户或实验保持稳定路由，以维持一致性与测量。 | Define identity, expiry, failover, policy override, privacy and experiment-analysis rules. 定义身份、过期、回退、策略覆盖、隐私与实验分析规则。 |
A governed router has four layers 受治理路由器有四层
Versioned model and endpoint capabilities, policy, price, region, health, lifecycle and provenance.
Deterministic filters that prevent optimization from selecting an illegal, unsupported or unsafe route.
Explicit objectives, weights, thresholds, normalization, tie-breaking, hysteresis and fallback.
Decision log, provider attempts, usage, cost, latency, quality, user result and reviewed policy promotion.
版本化模型与端点能力、策略、价格、区域、健康、生命周期与来源。
确定性过滤，防止优化选择非法、不支持或不安全路由。
明确目标、权重、阈值、标准化、平局规则、迟滞与回退。
决策日志、供应商尝试、用量、成本、延迟、质量、用户结果与经评审策略发布。
Build a routing benchmark 构建路由基准
- Label workload classes, hard constraints, accepted-quality thresholds, latency targets and maximum cost.
- Run pinned baselines for every model and provider endpoint before dynamic routing.
- Simulate outage, degradation, price change, capacity limit, policy change and evaluator drift.
- Canary by workload and require the trace to reproduce every eligible set, score and fallback.
- 标注负载类别、硬约束、可接受质量阈值、延迟目标与最高成本。
- 在动态路由前，为每个模型与供应商端点运行固定基线。
- 模拟中断、劣化、价格变化、容量限制、策略变化与评估器漂移。
- 按工作负载灰度，并要求调用链重现每个合格集合、评分与回退。
Separate eligibility from optimization 分离资格判断与优化
Eligibility is a deterministic, fail-closed gate over versioned metadata. Scoring operates only on the frozen eligible set. Execution adapters cannot silently widen that set. Online health can remove an endpoint immediately; adding or materially changing a route requires reviewed catalog and policy updates. The trace stores every input used by the decision.
资格判断是基于版本化元数据的确定性 Fail-Closed 门禁；评分只在冻结的合格集合上运行；执行适配器不得静默扩大集合。在线健康可立即移除端点，但新增或重大改变路由需经目录与策略评审更新。调用链存储决策使用的所有输入。
Production rule: routing is policy, not magic: the same frozen inputs and policy version should reproduce the same eligible set and decision.
生产规则：路由是策略，不是魔法：相同冻结输入与策略版本应重现相同合格集合与决策。
LLM routing chooses reasoning; QVeris chooses action LLM 路由选择推理，QVeris 选择行动
An LLM router selects how the agent reasons. QVeris then Discovers, Inspects and Calls the external API, tool, service or live dataset that can execute the intended action. Join their evidence in one workflow trace, but keep policies and retries separate.
LLM 路由器选择智能体如何推理；QVeris 随后发现、检查并调用能够执行预期行动的外部 API、工具、服务或实时数据集。两者证据连接在同一工作流调用链中，但策略与重试保持分离。
The Production Operating Model for LLM routingLLM 路由的生产运营模型
For LLM Routing, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“LLM 路由”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory hard eligibility constraints, task classification, candidate scoring, model and provider separation, uncertainty, exploration, fallback, budgets, policy overrides, and route explanations. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点硬性资格约束、任务分类、候选评分、模型与供应商分离、不确定性、探索、故障切换、预算、策略覆盖和路由解释。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For LLM Routing, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“LLM 路由”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate LLM Routing, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“LLM 路由”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating LLM Routing, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“LLM 路由”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
It selects a model, hosting provider or endpoint for a request using constraints, policy and operational signals.
Routing chooses before or during execution; fallback is a bounded reaction after an eligible failure.
Yes, when cheaper eligible routes maintain accepted quality and savings exceed retries, review, platform and engineering cost.
使用约束、策略与运行信号，为请求选择模型、托管供应商或端点。
路由在执行前或执行中选择；回退是在符合条件失败后的有边界反应。
可以，前提是更便宜合格路由保持可接受质量，且节省超过重试、审核、平台与工程成本。