LLM Load Balancing
Balance Equivalent Endpoints, Not Labels
LLM 负载均衡：只在能力等价的端点间分配请求
LLM load balancing distributes eligible requests across deployments or providers. The eligibility filter matters more than the algorithm: every candidate must preserve the workload contract.
LLM 负载均衡在合格的 Deployment 或供应商之间分配请求。合格过滤比算法更重要：每个候选都必须保留工作负载契约。
TL;DR
Filter by model capability, context, tools, region, policy and quota before balancing.
Weights, load, latency, quota and health answer different operational questions.
Stale health, rate-limit or concurrency data can route traffic into failure.
Protect tenants, priorities and budgets instead of letting one workload consume the pool.
均衡前按模型能力、上下文、工具、区域、策略与配额过滤。
权重、负载、延迟、配额与健康度回答不同运营问题。
过期健康、限流或并发数据会把流量导向故障。
保护租户、优先级与预算，避免单一负载耗尽资源池。
Load balancing is constrained selection 负载均衡是受约束选择
Traditional balancing assumes interchangeable servers behind one service. LLM endpoints are often not interchangeable: they may differ in model version, context window, tools, modalities, data region, price, quotas and output behavior. Create pools only after declaring equivalence for the workload.
传统均衡假设同一服务后的服务器可互换；LLM 端点往往不可互换，可能在模型版本、上下文、工具、模态、数据区域、价格、配额与输出行为上不同。必须先为工作负载声明等价性，再建立资源池。
Within an eligible pool, the selector can use static weights, active concurrency, observed latency, remaining quota, recent error rate or combinations. Circuit breakers remove unhealthy endpoints; recovery probes return them gradually.
在合格池内，选择器可以使用静态权重、活跃并发、观测延迟、剩余配额、近期错误率或组合信号。熔断器移除不健康端点，恢复探针再让其逐步回归。
Common load-balancing strategies 常见负载均衡策略
| Strategy 策略 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Weighted distribution 加权分配 | Stable capacity or commercial allocation across equivalent endpoints. 在等价端点间按稳定容量或商业分配流量。 | Weights require maintenance and do not react to sudden degradation alone. 权重需要维护，单独使用时无法响应突发退化。 |
| Least loaded 最少负载 | Pools where active concurrency predicts queueing. 活跃并发能够预测排队的资源池。 | Define comparable load units and avoid oscillation from noisy samples. 定义可比负载单位，并避免噪声样本导致振荡。 |
| Latency aware 延迟感知 | Interactive workloads with measured regional paths. 具有实测区域路径的交互式负载。 | Use workload-specific tail latency and protect against cold-start bias. 使用工作负载特定尾延迟，并防止冷启动偏差。 |
| Quota aware 配额感知 | Multiple keys, deployments or providers with independent limits. 拥有独立限制的多个密钥、Deployment 或供应商。 | Respect tenant budgets and native reset signals; do not mask exhaustion indefinitely. 尊重租户预算与原生重置信号，不能无限掩盖耗尽。 |
| Health driven 健康驱动 | Rapid isolation of failing or degraded endpoints. 快速隔离故障或退化端点。 | Tune trip, cooldown and recovery probes to avoid flapping. 调优触发、冷却与恢复探针，避免抖动。 |
Build an equivalent endpoint pool 构建能力等价端点池
Version model, context, tools, schemas, modalities and stream behavior.
Track health, concurrency, quota, latency, circuit state and data timestamp.
Reserve capacity, set priority and cap concurrency or spend per identity.
Record candidate set, exclusions, signals, strategy version and selected endpoint.
版本化模型、上下文、工具、结构定义、模态与流行为。
跟踪健康、并发、配额、延迟、熔断状态与数据时间。
按身份预留容量、设置优先级并限制并发或成本。
记录候选集合、排除项、信号、策略版本与所选端点。
Load-test the routing policy 压测路由策略
- Replay representative workloads with capability and tenant labels.
- Inject slow, rate-limited, unhealthy and quota-exhausted endpoint states.
- Measure tail latency, queueing, error rate, distribution and fairness by workload.
- Verify circuit trip, cooldown, recovery probes and gradual re-entry.
- Confirm no request is routed to an endpoint missing a required capability.
- 使用带能力与租户标签的代表性负载回放。
- 注入缓慢、限流、不健康与配额耗尽端点状态。
- 按工作负载测量尾延迟、排队、错误率、分布与公平性。
- 验证熔断触发、冷却、恢复探针与渐进回归。
- 确认任何请求都不会路由到缺失必需能力的端点。
Filter hard constraints, then optimize 先过滤硬约束，再优化
A versioned catalog expands the workload alias into candidate endpoints. Hard filters remove candidates that fail capability, policy, region, tenant or quota rules. The selector scores the remaining pool from fresh operational state. Execution updates attempt evidence and health signals, while circuit breakers isolate faults outside the critical decision path.
版本化目录把工作负载别名展开为候选端点。硬性过滤器移除不满足能力、策略、区域、租户或配额规则的候选；选择器再根据新鲜运营状态为剩余资源池评分。执行更新尝试证据与健康信号，熔断器在关键决策路径之外隔离故障。
Production rule: never balance across endpoints merely because they share a model alias.
生产规则：绝不能仅因端点共享模型别名就把它们放进同一均衡池。
Balance models and external capabilities independently 分别均衡模型与外部能力
An AI gateway distributes inference traffic. QVeris complements it by discovering and calling external APIs, tools, services and live data. Share workload identity and trace context, but keep health, quotas, retries and fallback policies separate for model endpoints and external capabilities.
AI 网关分配推理流量；QVeris 补充外部 API、工具、服务与实时数据的发现和调用。两层共享工作负载身份与调用链上下文，但模型端点与外部能力应分别维护健康、配额、重试与故障切换策略。
The Production Operating Model for LLM load balancingLLM 负载均衡的生产运营模型
For LLM Load Balancing, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“LLM 负载均衡”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory capability eligibility, health, region, quotas, concurrency, queue depth, latency, cost, quality, session affinity, cache locality, fairness, and provider concentration. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点能力资格、健康、区域、配额、并发、队列深度、延迟、成本、质量、会话亲和、缓存局部性、公平性和供应商集中度。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For LLM Load Balancing, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“LLM 负载均衡”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate LLM Load Balancing, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“LLM 负载均衡”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating LLM Load Balancing, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“LLM 负载均衡”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
Only when the workload explicitly accepts their capability and quality differences.
There is no universal winner; select signals that match the bottleneck and test combinations.
No. Balancing selects among healthy eligible endpoints; fallback reacts to a failed attempt or unavailable preference.
只有工作负载明确接受其能力与质量差异时才可以。
没有万能赢家；应选择匹配瓶颈的信号并测试组合。
不等于。均衡在健康合格端点间选择；故障切换响应失败尝试或首选不可用。