Load Balance LLM Providers
Group by Capability Before Traffic跨 LLM 供应商负载均衡：分配流量前先按能力分组
Provider load balancing works only inside a capability-equivalent pool. Catalog endpoints, filter hard constraints, choose fresh signals and test fairness and failure behavior under real load.
供应商负载均衡只能在能力等价池内工作。目录化端点、过滤硬约束、选择新鲜信号，并在真实负载下测试公平与故障行为。
TL;DR
Same model label does not guarantee equivalent context, tools, region or policy.
Weights, concurrency, latency, quota and health optimize different problems.
Use quotas, priority and fair queues so one workload cannot consume the pool.
A balanced happy path says nothing about circuit breakers or recovery.
相同模型标签不保证上下文、工具、区域或策略等价。
权重、并发、延迟、配额与健康优化不同问题。
用配额、优先级与公平队列避免单一负载耗尽资源池。
正常路径均衡并不能证明熔断或恢复有效。
How to load balance LLM providers: catalog to canary如何在多个 LLM 供应商间负载均衡：从目录到灰度发布
Register every endpoint with an immutable model version, supported input and output modes, context limits, tool and structured-output behavior, region, policy, quotas and health source. Do not place endpoints in one pool merely because their marketing names look similar. Create workload-specific pools only after the same contract fixtures pass.
为每个端点登记不可变模型版本、输入与输出模态、上下文限制、工具和结构化输出行为、区域、策略、配额及健康来源。不能因为营销名称相似就把端点放入同一资源池；只有通过同一组契约 Fixture 后，才可建立面向具体工作负载的资源池。
Then choose the simplest strategy that matches the workload. Round robin may suit genuinely equivalent batch tasks; least-loaded selection needs trustworthy in-flight counters; latency-aware routing needs fresh regional measurements and a guard against sending all traffic to a temporarily fast endpoint. Add circuit breakers, replay realistic traffic, inject failures and canary policy changes before expanding.
随后选择与工作负载相符的最简单策略。轮询适合真正等价的批处理任务；最少负载策略依赖可信的进行中请求计数；延迟感知路由需要新鲜的区域测量，还要防止流量全部涌向暂时较快的端点。增加熔断，回放真实流量、注入故障，并在扩大范围前灰度发布策略变更。
Setup choices设置选择
| Choice选择 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Weighted加权 | Stable allocations across equivalent capacity.在等价容量间进行稳定分配。 | Maintain weights and combine with health removal.维护权重，并与健康移除结合。 |
| Least loaded最少负载 | Concurrency predicts queueing for the workload.并发能够预测该负载的排队。 | Define comparable load units and damp noisy changes.定义可比负载单位并抑制噪声变化。 |
| Latency aware延迟感知 | Interactive paths have trustworthy regional measurements.交互路径具有可信区域测量。 | Use tail latency and avoid cold-start bias.使用尾延迟并避免冷启动偏差。 |
| Quota aware配额感知 | Endpoints have independent limits or reset windows.端点拥有独立限制或重置窗口。 | Respect tenant budgets and native retry signals.尊重租户预算与原生重试信号。 |
Operational safeguards运营保护
Timestamp health, load, quota and circuit data used by each decision.
Define trip, cooldown, half-open probes and gradual re-entry.
Cap tenant concurrency and spend; reserve priority capacity deliberately.
Store candidate set, exclusions, signal snapshot, strategy and result.
为每次决策使用的健康、负载、配额与熔断数据标记时间。
定义触发、冷却、半开探针与渐进回归。
限制租户并发与成本，并有意预留优先容量。
存储候选集、排除项、信号快照、策略与结果。
Validate under realistic load在真实负载下验证
A uniform request generator can hide the conditions that break a production router. Build a replay set from actual prompt sizes, output lengths, streaming duration, tool use and tenant distribution. Keep expected outputs or validators with the replay so the test detects semantic drift as well as transport failure.
均匀的压测请求很容易掩盖生产路由真正会遇到的问题。应根据实际 Prompt 长度、输出长度、流式持续时间、工具使用和租户分布构建回放集，并为回放保留期望输出或验证器，让测试既能发现传输失败，也能识别语义漂移。
- Replay workload mixes, bursts, long streams and noisy-neighbor tenant contention.
- Inject regional latency, 429, 5xx, quota exhaustion and stale or contradictory health signals.
- Measure request distribution, queue age, time to first token, tail latency, errors, cost and fairness by workload.
- Verify a long stream does not occupy all capacity reserved for short interactive requests.
- Canary policy changes by workload and retain a pinned-route rollback with the prior registry version.
- 回放负载组合、突发、长流以及“吵闹邻居”式租户竞争。
- 注入区域延迟、429、5xx、配额耗尽，以及过期或相互矛盾的健康信号。
- 按工作负载测量请求分布、队列年龄、首 Token 延迟、尾延迟、错误、成本与公平性。
- 验证长流式请求不会占满为短交互请求保留的全部容量。
- 按工作负载灰度策略变更，并保留携带旧注册表版本的固定路由回滚。
Filter, score, dispatch, observe过滤、评分、派发、观测
A versioned catalog produces candidates. Hard filters first remove endpoints that fail mandatory requirements such as modality, context size, tool support, region, tenant policy or remaining quota. The selector then scores only the eligible set using fresh latency, capacity, cost and error signals. This ordering matters: a low score must never make an incompatible route eligible.
版本化目录先产生候选项。硬过滤器首先剔除不满足模态、上下文大小、工具支持、区域、租户策略或剩余配额等强制要求的端点；选择器再根据新鲜的延迟、容量、成本和错误信号，只对合格集合评分。顺序非常重要：再高的分数也不能让不兼容路由变成合格路由。
Dispatch one bounded attempt and record the catalog version, candidate set, rejected reasons, score inputs and selected route. Update health and circuit-breaker state outside the critical path, but attach freshness timestamps so stale telemetry cannot masquerade as current capacity. A fallback consumes the same parent deadline and attempt budget rather than starting a new request lifecycle.
派发一次有界尝试，并记录目录版本、候选集合、淘汰原因、评分输入和最终路由。健康状态与熔断器可在关键路径之外更新，但必须附带新鲜度时间戳，避免过期遥测被误认为当前容量。故障切换继续消耗同一个父请求截止时间和尝试预算，而不是启动新的请求生命周期。
Production rule: balance only among endpoints proven equivalent for the workload, and keep an explicit pinned route for diagnosis and rollback.
生产规则：只在已证明对该工作负载等价的端点间均衡，并保留明确的固定路由用于诊断与回滚。
Balance inference and capabilities independently分别均衡推理与能力
The gateway balances model endpoints, while QVeris discovers and calls external APIs, tools, services and live data. These are different availability domains. A healthy model route does not prove that a market-data source, search tool or transaction API is healthy, entitled for the tenant or fresh enough for the task.
网关负责均衡模型端点，而 QVeris 发现并调用外部 API、工具、服务与实时数据。二者属于不同的可用性域：模型路由健康，并不能证明市场数据源、搜索工具或交易 API 同样健康，也不能证明租户拥有权限或数据足够新鲜。
Keep model and capability pools, quotas, circuit breakers and retry budgets independent. Share only the context needed to correlate the workflow—tenant, parent trace, deadline, region and policy snapshot. If inference falls back after a tool result has already been produced, pass the verified result forward rather than calling the tool again unless its freshness contract requires a refresh.
模型与能力应分别维护资源池、配额、熔断器和重试预算，只共享关联工作流所需的租户、父 Trace、截止时间、区域和策略快照。如果工具结果已经产生后推理才发生故障切换，应继续传递已验证结果；只有数据新鲜度契约明确要求刷新时，才再次调用工具。
Use healthy eligible routes, then balance先筛选健康且合格的路由，再做负载均衡
Load balancing must not send requests to a cheap route that lacks required features or data policy. Filter by capability, region, tenant, and current health; then use a stable selection method so retries do not randomly fan out.
负载均衡不能把请求发给缺少所需功能或数据策略的低价路由。先按能力、区域、租户与当前健康状态过滤，再使用稳定选择方法，避免重试随机扩散。
def choose_route(request, routes):
eligible = [r for r in routes
if r.healthy
and request.capabilities <= r.capabilities
and request.region in r.allowed_regions]
if not eligible:
raise NoEligibleRoute(request.capabilities, request.region)
bucket = stable_hash(request.tenant_id, request.request_id) % sum(r.weight for r in eligible)
for route in eligible:
bucket -= route.weight
if bucket < 0:
return route
raise AssertionError("unreachable")- Update health from bounded windows, not one isolated error.
- Cap per-route concurrency and remove a route gradually to avoid traffic shocks.
- Measure accepted quality, latency percentiles, errors, and cost by workload—not globally.
- 健康状态应来自有界窗口，而不是单次错误。
- 限制每条路由并发，并逐步摘除，避免流量冲击。
- 按工作负载而不是全局测量合格质量、延迟分位、错误与成本。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
FAQ
Only if the workload contract accepts their capability and quality differences.
A simple weighted or round-robin policy plus strict health removal is easier to validate.
It is related but distinct: balancing chooses before execution; fallback reacts after failure.
只有工作负载契约接受其能力与质量差异时才可以。
简单加权或轮询加严格健康移除更易验证。
相关但不同：均衡在执行前选择，故障切换在失败后响应。