Load Balance LLM Providers
Group by Capability Before Traffic
跨 LLM 供应商负载均衡：分配流量前先按能力分组
Provider load balancing works only inside a capability-equivalent pool. Catalog endpoints, filter hard constraints, choose fresh signals and test fairness and failure behavior under real load.
供应商负载均衡只能在能力等价池内工作。目录化端点、过滤硬约束、选择新鲜信号，并在真实负载下测试公平与故障行为。
TL;DR
Same model label does not guarantee equivalent context, tools, region or policy.
Weights, concurrency, latency, quota and health optimize different problems.
Use quotas, priority and fair queues so one workload cannot consume the pool.
A balanced happy path says nothing about circuit breakers or recovery.
相同模型标签不保证上下文、工具、区域或策略等价。
权重、并发、延迟、配额与健康优化不同问题。
用配额、优先级与公平队列避免单一负载耗尽资源池。
正常路径均衡并不能证明熔断或恢复有效。
How to load balance LLM providers: catalog to canary 如何在多个 LLM 供应商间负载均衡：从目录到灰度发布
Register every endpoint with model version, capabilities, region, policy, quotas and health source. Create workload-specific pools only after contract tests pass.
为每个端点注册模型版本、能力、区域、策略、配额与健康来源。只有契约测试通过后，才建立工作负载特定资源池。
Select a conservative strategy, add circuit breakers and evidence, replay realistic traffic, inject failures and canary before expanding.
选择保守策略，增加熔断与证据，回放真实流量、注入故障并灰度后再扩展。
Setup choices 设置选择
| Choice 选择 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Weighted 加权 | Stable allocations across equivalent capacity. 在等价容量间进行稳定分配。 | Maintain weights and combine with health removal. 维护权重，并与健康移除结合。 |
| Least loaded 最少负载 | Concurrency predicts queueing for the workload. 并发能够预测该负载的排队。 | Define comparable load units and damp noisy changes. 定义可比负载单位并抑制噪声变化。 |
| Latency aware 延迟感知 | Interactive paths have trustworthy regional measurements. 交互路径具有可信区域测量。 | Use tail latency and avoid cold-start bias. 使用尾延迟并避免冷启动偏差。 |
| Quota aware 配额感知 | Endpoints have independent limits or reset windows. 端点拥有独立限制或重置窗口。 | Respect tenant budgets and native retry signals. 尊重租户预算与原生重试信号。 |
Operational safeguards 运营保护
Timestamp health, load, quota and circuit data used by each decision.
Define trip, cooldown, half-open probes and gradual re-entry.
Cap tenant concurrency and spend; reserve priority capacity deliberately.
Store candidate set, exclusions, signal snapshot, strategy and result.
为每次决策使用的健康、负载、配额与熔断数据标记时间。
定义触发、冷却、半开探针与渐进回归。
限制租户并发与成本，并有意预留优先容量。
存储候选集、排除项、信号快照、策略与结果。
Validate under realistic load 在真实负载下验证
- Replay workload mixes, bursts, long streams and tenant contention.
- Inject latency, 429, 5xx, quota exhaustion and stale health.
- Measure distribution, tail latency, errors and fairness by workload.
- Canary policy changes and retain a pinned-route rollback.
- 回放负载组合、突发、长流与租户竞争。
- 注入延迟、429、5xx、配额耗尽与过期健康。
- 按负载测量分布、尾延迟、错误与公平。
- 灰度策略变更，并保留固定路由回滚。
Filter, score, dispatch, observe 过滤、评分、派发、观测
A catalog produces candidates. Hard filters remove incompatible endpoints. The selector scores the remainder from fresh state, dispatches one request and records the decision. Circuit breakers update availability outside the critical path.
目录产生候选；硬过滤器移除不兼容端点；选择器基于新鲜状态为剩余项评分、派发一次请求并记录决策；熔断器在关键路径之外更新可用性。
Production rule: balance only among endpoints proven equivalent for the workload.
生产规则：只在已证明对该工作负载等价的端点间均衡。
Balance inference and capabilities independently 分别均衡推理与能力
The gateway balances model endpoints. QVeris discovers and calls external APIs, tools, services and live data. Keep their pools, quotas and retries independent while sharing tenant and trace context.
网关均衡模型端点；QVeris 发现并调用外部 API、工具、服务与实时数据。两者资源池、配额与重试应独立，同时共享租户与调用链上下文。
Use healthy eligible routes, then balance 先筛选健康且合格的路由，再做负载均衡
Load balancing must not send requests to a cheap route that lacks required features or data policy. Filter by capability, region, tenant, and current health; then use a stable selection method so retries do not randomly fan out.
负载均衡不能把请求发给缺少所需功能或数据策略的低价路由。先按能力、区域、租户与当前健康状态过滤，再使用稳定选择方法，避免重试随机扩散。
def choose_route(request, routes):
eligible = [r for r in routes
if r.healthy
and request.capabilities <= r.capabilities
and request.region in r.allowed_regions]
if not eligible:
raise NoEligibleRoute(request.capabilities, request.region)
bucket = stable_hash(request.tenant_id, request.request_id) % sum(r.weight for r in eligible)
for route in eligible:
bucket -= route.weight
if bucket < 0:
return route
raise AssertionError("unreachable")
- Update health from bounded windows, not one isolated error.
- Cap per-route concurrency and remove a route gradually to avoid traffic shocks.
- Measure accepted quality, latency percentiles, errors, and cost by workload—not globally.
- 健康状态应来自有界窗口，而不是单次错误。
- 限制每条路由并发，并逐步摘除，避免流量冲击。
- 按工作负载而不是全局测量合格质量、延迟分位、错误与成本。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
The Production Operating Model for load balancing across LLM providers跨 LLM 供应商负载均衡的生产运营模型
For Load Balance LLM Providers, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“均衡 LLM 供应商”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory provider-specific model identity, equivalent capability pools, regional endpoints, quota windows, health signals, latency distributions, error classes, cost, and contract limits. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点供应商专属模型身份、等价能力池、区域端点、配额窗口、健康信号、延迟分布、错误类别、成本和合同限制。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For Load Balance LLM Providers, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“均衡 LLM 供应商”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate Load Balance LLM Providers, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“均衡 LLM 供应商”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating Load Balance LLM Providers, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“均衡 LLM 供应商”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
Only if the workload contract accepts their capability and quality differences.
A simple weighted or round-robin policy plus strict health removal is easier to validate.
It is related but distinct: balancing chooses before execution; fallback reacts after failure.
只有工作负载契约接受其能力与质量差异时才可以。
简单加权或轮询加严格健康移除更易验证。
相关但不同：均衡在执行前选择，故障切换在失败后响应。