LLM Load Balancing
Balance Equivalent Endpoints, Not LabelsLLM 负载均衡：只在能力等价的端点间分配请求
LLM load balancing distributes eligible requests across deployments or providers. The eligibility filter matters more than the algorithm: every candidate must preserve the workload contract.
LLM 负载均衡在合格的 Deployment 或供应商之间分配请求。合格过滤比算法更重要：每个候选都必须保留工作负载契约。
TL;DR
Filter by model capability, context, tools, region, policy and quota before balancing.
Weights, load, latency, quota and health answer different operational questions.
Stale health, rate-limit or concurrency data can route traffic into failure.
Protect tenants, priorities and budgets instead of letting one workload consume the pool.
均衡前按模型能力、上下文、工具、区域、策略与配额过滤。
权重、负载、延迟、配额与健康度回答不同运营问题。
过期健康、限流或并发数据会把流量导向故障。
保护租户、优先级与预算，避免单一负载耗尽资源池。
Load balancing is constrained selection负载均衡是受约束选择
Traditional balancing assumes interchangeable servers behind one service. LLM endpoints are often not interchangeable: they may differ in immutable model version, context window, tool and structured-output behavior, modality, data region, price, quota and response semantics. Define equivalence for a specific workload before creating a pool; a route that is excellent for summarization may still be ineligible for vision or tool-heavy agents.
传统负载均衡假设同一服务后的服务器可以互换；LLM 端点却常在不可变模型版本、上下文窗口、工具与结构化输出行为、模态、数据区域、价格、配额和响应语义上存在差异。建立资源池前，应先针对具体工作负载定义等价性；适合摘要的优质路由，仍可能不适合视觉或重工具 Agent。
Separate hard eligibility from optimization. Capability, policy, region, tenant entitlement and remaining context are pass-or-fail gates. Only after those gates pass should the selector use static weights, in-flight concurrency, observed latency, remaining quota, recent error rate, cost or a combination. This prevents a cheap or fast endpoint from winning a request it cannot safely complete.
硬性合格判断与优化评分必须分开。能力、策略、区域、租户权限和剩余上下文属于通过或不通过的门禁；只有门禁通过后，选择器才使用静态权重、进行中并发、观测延迟、剩余配额、近期错误率、成本或其组合评分，从而避免便宜或快速却无法安全完成任务的端点胜出。
Circuit breakers remove unhealthy endpoints, but health must be workload-aware. A provider can serve short text normally while long streams or tools are degraded. Track health by endpoint and capability, attach freshness timestamps, and use limited recovery probes plus gradual traffic re-entry instead of immediately restoring full weight.
熔断器负责移除不健康端点，但健康状态也要感知工作负载。某供应商可能仍能正常处理短文本，却在长流或工具调用上退化。应按端点与能力记录健康状态并附带新鲜度时间戳，再通过有限恢复探针和渐进回流，而不是立即恢复全部权重。
Common load-balancing strategies常见负载均衡策略
| Strategy策略 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Weighted distribution加权分配 | Stable capacity or commercial allocation across equivalent endpoints.在等价端点间按稳定容量或商业分配流量。 | Weights require maintenance and do not react to sudden degradation alone.权重需要维护，单独使用时无法响应突发退化。 |
| Least loaded最少负载 | Pools where active concurrency predicts queueing.活跃并发能够预测排队的资源池。 | Define comparable load units and avoid oscillation from noisy samples.定义可比负载单位，并避免噪声样本导致振荡。 |
| Latency aware延迟感知 | Interactive workloads with measured regional paths.具有实测区域路径的交互式负载。 | Use workload-specific tail latency and protect against cold-start bias.使用工作负载特定尾延迟，并防止冷启动偏差。 |
| Quota aware配额感知 | Multiple keys, deployments or providers with independent limits.拥有独立限制的多个密钥、Deployment 或供应商。 | Respect tenant budgets and native reset signals; do not mask exhaustion indefinitely.尊重租户预算与原生重置信号，不能无限掩盖耗尽。 |
| Health driven健康驱动 | Rapid isolation of failing or degraded endpoints.快速隔离故障或退化端点。 | Tune trip, cooldown and recovery probes to avoid flapping.调优触发、冷却与恢复探针，避免抖动。 |
Build an equivalent endpoint pool构建能力等价端点池
Version model, context, tools, schemas, modalities and stream behavior.
Track health, concurrency, quota, latency, circuit state and data timestamp.
Reserve capacity, set priority and cap concurrency or spend per identity.
Record candidate set, exclusions, signals, strategy version and selected endpoint.
版本化模型、上下文、工具、结构定义、模态与流行为。
跟踪健康、并发、配额、延迟、熔断状态与数据时间。
按身份预留容量、设置优先级并限制并发或成本。
记录候选集合、排除项、信号、策略版本与所选端点。
Load-test the routing policy压测路由策略
Test the policy with production-shaped traffic rather than equal-size synthetic prompts. Replays should preserve prompt length, output length, stream duration, tool use, tenant mix and burst patterns. Include validators so a distribution that looks operationally healthy cannot hide lower-quality or invalid outputs.
应使用接近生产形态的流量测试策略，而不是大小相同的合成 Prompt。回放需要保留 Prompt 长度、输出长度、流持续时间、工具使用、租户组合与突发模式，并加入验证器，防止运营指标看似健康的分布掩盖质量下降或无效输出。
- Replay representative workloads with explicit capability, region, policy and tenant labels.
- Inject slow, rate-limited, unhealthy, quota-exhausted and stale-health endpoint states.
- Measure queue age, time to first token, tail latency, errors, cost, distribution and fairness by workload.
- Verify circuit trip, cooldown, probe limits and gradual re-entry without a traffic surge.
- Confirm no request reaches an endpoint missing a required capability or tenant permission.
- Canary a new policy and retain a pinned-route rollback to the previous catalog version.
- 使用带明确能力、区域、策略与租户标签的代表性工作负载回放。
- 注入缓慢、限流、不健康、配额耗尽与健康状态过期的端点。
- 按工作负载测量队列年龄、首 Token 延迟、尾延迟、错误、成本、分布与公平性。
- 验证熔断、冷却、探针限制与渐进回流不会引发流量突增。
- 确认任何请求都不会进入缺失必需能力或租户权限的端点。
- 灰度新策略，并保留固定路由到旧目录版本的回滚能力。
Filter hard constraints, then optimize先过滤硬约束，再优化
A versioned catalog expands the workload alias into candidate endpoints and adapters. Hard filters remove candidates that fail capability, policy, region, tenant, context or quota rules. The selector scores only the remaining pool from fresh operational state. Store the candidate set and rejection reasons, not just the winner, so operators can explain why an apparently healthy route was excluded.
版本化目录把工作负载别名展开为候选端点与 Adapter。硬过滤器移除不满足能力、策略、区域、租户、上下文或配额规则的候选；选择器只根据新鲜运营状态为剩余池评分。除了最终胜出项，还应保存候选集合与淘汰原因，让运维人员能够解释看似健康的路由为什么被排除。
Dispatch one bounded attempt under a parent deadline and attempt budget. Execution records catalog version, score inputs, selected route, native request ID, outcome and usage. Health aggregation and circuit-breaker updates can run outside the critical path, but their data must carry timestamps and sample size. A fallback re-enters the same state machine with the failed route excluded.
在父截止时间与尝试预算下派发一次有界尝试，并记录目录版本、评分输入、所选路由、原生 Request ID、结果与用量。健康聚合和熔断器更新可以在关键路径之外运行，但数据必须附带时间戳与样本量。故障切换继续进入同一状态机，并排除已经失败的路由。
Production rule: never balance across endpoints merely because they share a model alias; equivalence must be proven per workload.
生产规则：绝不能仅因端点共享模型别名就放入同一均衡池；等价性必须按工作负载证明。
Balance models and external capabilities independently分别均衡模型与外部能力
An AI gateway distributes inference traffic. QVeris complements it by discovering and calling external APIs, tools, services and live data. These are separate availability domains: a healthy model does not prove that a data source is fresh, a tool is entitled for the tenant or an external action is safe to repeat.
AI Gateway 分配推理流量；QVeris 补充外部 API、工具、服务与实时数据的发现和调用。二者属于不同可用性域：模型健康并不能证明数据源足够新鲜、租户拥有工具权限，或外部动作可以安全重复。
Share workload identity, parent trace, deadline, region and policy snapshot, but keep health, quotas, circuit breakers, retries and fallback pools independent. If inference is retried after a capability result has been produced, reuse the verified result unless its freshness contract requires another call, and retain a separate action ID for every external side effect.
两层可共享工作负载身份、父 Trace、截止时间、区域与策略快照，但健康、配额、熔断器、重试和故障池应分别维护。如果能力结果产生后推理才重试，应复用已验证结果；只有新鲜度契约要求时才再次调用，并为每个外部副作用保留独立 Action ID。
FAQ
Only when the workload explicitly accepts their capability and quality differences.
There is no universal winner; select signals that match the bottleneck and test combinations.
No. Balancing selects among healthy eligible endpoints; fallback reacts to a failed attempt or unavailable preference.
只有工作负载明确接受其能力与质量差异时才可以。
没有万能赢家；应选择匹配瓶颈的信号并测试组合。
不等于。均衡在健康合格端点间选择；故障切换响应失败尝试或首选不可用。