LLM High Availability
Design Failure Domains, Prove RecoveryLLM 高可用：设计故障域，并用演练证明可恢复

TL;DR

Availability starts with workload contracts高可用始于工作负载契约

For each workload, define the required model capabilities, maximum latency, data region, policy, output validation and acceptable degraded modes. An endpoint belongs in the failover pool only after it passes that contract.

为每项工作负载定义所需模型能力、最大延迟、数据区域、策略、输出验证和可接受的降级方式。端点只有通过这份契约后，才能进入故障切换资源池。

Translate the contract into measurable service objectives. A conversational assistant may prioritize time to first token and tolerate a smaller fallback model, while a document extraction job may accept queueing but require the same schema and accuracy. A regulated workflow may allow no cross-region failover at all. “Any model response” is therefore not a valid availability target.

还要把契约转化为可测量的服务目标。对话助手可能优先关注首 Token 延迟，并能接受能力较小的备用模型；文档提取任务可以接受排队，却必须保持相同的输出结构和准确率；受监管工作流甚至可能完全禁止跨区域切换。因此，“只要有模型返回即可”并不是有效的高可用目标。

List correlated risks. Two provider labels may share a cloud region, network, gateway, identity system or configuration source. Redundancy counts only when the relevant failure domains are independent.

必须列出相关性风险。两个看似不同的供应商可能共享云区域、网络出口、网关、身份系统或配置来源。只有与目标故障场景相关的故障域彼此独立时，冗余才真正成立。

Build a dependency map for DNS, certificate issuance, identity, secrets, policy distribution, telemetry and outbound networking as well as the model providers themselves. If every route depends on one authorization service or one configuration database, adding a third model provider does not remove the main single point of failure.

依赖图不仅要包含模型供应商，还要覆盖 DNS、证书签发、身份认证、密钥、策略分发、遥测和出口网络。如果所有路由仍依赖同一个授权服务或配置数据库，那么即使增加第三家模型供应商，也没有消除真正的单点故障。

Availability pattern decisions高可用模式决策

High-availability controls高可用控制

Run recovery game days开展恢复演练

A failover configuration that has never handled a controlled failure is only a hypothesis. Schedule game days with clear success criteria: detection time, route-decision time, percentage of requests completed within deadline, output-validity rate, recovery-point objective for queued work and the time required to return to normal routing.

从未处理过受控故障的切换配置只能算一种假设。应定期开展恢复演练，并明确成功标准：故障检测时间、路由决策时间、在截止时间内完成的请求比例、输出有效率、排队任务的恢复点目标，以及恢复正常路由所需时间。

• Fail one gateway, provider, model, region and configuration dependency at a time.
• Inject correlated failures, slow responses, broken streams and stale health.
• Verify deadlines, idempotency, tenant fairness, data policy and controlled errors.
• Exercise failback and rollback so recovery does not create a second incident.

• 依次让网关、供应商、模型、区域和配置依赖发生故障。
• 注入相关性故障、慢响应、流式中断和过期健康状态。
• 验证截止时间、幂等、租户公平性、数据策略与受控错误。
• 演练恢复切回和回滚，避免恢复过程制造第二次事故。

Do not limit exercises to hard outages. Rate limits, elevated latency, malformed structured output and streams that stall after the first token are more common and often harder to classify. Test whether the gateway spends the entire deadline on the first route, whether retries amplify provider pressure and whether a fallback response still satisfies the workload contract.

演练不能只覆盖完全宕机。速率限制、延迟升高、结构化输出失效，以及输出首个 Token 后卡住的流式响应更常见，也更难判断。需要验证网关是否把全部截止时间浪费在第一条路由上、重试是否进一步放大供应商压力，以及备用响应是否仍满足工作负载契约。

Failback deserves its own test. A provider that has just recovered may be unstable or capacity-constrained, so restore traffic gradually with health hysteresis and canaries. Compare error, latency and validation metrics before returning the route to full weight, and preserve the option to reverse the change without interrupting in-flight requests.

恢复切回也需要独立测试。刚恢复的供应商可能仍不稳定或容量受限，因此应通过健康状态滞后机制和灰度流量逐步恢复。重新赋予完整流量权重前，要比较错误率、延迟和验证指标，并保留不打断在途请求即可撤销变更的能力。

Separate control and data planes分离控制面与数据面

The control plane manages versioned catalogs, policies, health and rollout. The data plane validates each request, filters by capability and residency, selects a healthy route and executes a bounded attempt. Both planes are replicated independently and emit joined evidence.

控制面管理版本化目录、策略、健康状态和发布；数据面验证每个请求，按能力和数据驻留要求过滤，选择健康路由并执行有边界的尝试。两个平面独立复制，同时生成可关联的证据。

The request path should continue operating for a bounded period when the control plane is unavailable. Data-plane replicas can use signed configuration snapshots and last-known-good policy, but every snapshot needs a version, expiry and explicit behavior when it becomes too old. High-risk policy changes and emergency revocations require a faster invalidation channel than ordinary catalog updates.

控制面不可用时，请求路径仍应在有限时间内继续工作。数据面副本可以使用已签名配置快照和最近一次有效策略，但每份快照都需要版本、到期时间，以及过期后的明确处理方式。高风险策略变更和紧急吊销必须拥有比普通目录更新更快的失效通道。

Route selection needs bounded attempts. Allocate the end-to-end deadline across connection, first-token and completion phases; reserve time for at most the planned number of fallbacks; and stop retrying when the remaining budget cannot produce a useful result. Hedging can reduce tail latency for selected idempotent workloads, but uncontrolled duplicate requests increase cost and can worsen a provider incident.

路由选择必须限制尝试次数。应把端到端截止时间分配给连接、首 Token 和完整生成阶段，为计划中的故障切换预留时间；当剩余时间不足以生成有用结果时立即停止重试。对部分幂等工作负载，对冲请求可以降低尾部延迟，但不受控的重复请求会增加成本，并可能加剧供应商事故。

Design tool availability as a separate system把工具可用性设计为独立系统

Model availability restores inference. QVeris governs external APIs, tools, services and live data. Give those capabilities separate pools, timeouts, idempotency and recovery tests, then connect both through the workflow trace.

模型高可用解决的是推理恢复；QVeris 治理外部 API、工具、服务和实时数据。应为这些能力配置独立的资源池、超时、幂等和恢复测试，再通过工作流调用链把两层关联起来。

A model fallback cannot repair an unavailable payment API, stale market-data feed or document parser. Define availability and degradation for each capability: whether cached data is acceptable, how old it may be, whether a read-only substitute exists and which actions must stop rather than retry. For side-effecting tools, preserve idempotency keys across model and provider retries so recovery does not duplicate an order or message.

模型故障切换无法修复不可用的支付 API、过期的市场数据流或失效的文档解析器。每项能力都应单独定义可用性与降级规则：能否使用缓存数据、允许多旧、是否有只读替代方案，以及哪些动作必须停止而不能重试。对带副作用的工具，还要在模型重试和供应商重试之间保留幂等键，避免恢复过程中重复提交订单或消息。

Use the shared workflow trace to distinguish model, routing and tool failures. Record the model attempt, capability selected, provider-native request ID, timeout phase, validation result and any recovery action. Operators then see whether the user received a degraded but valid answer, a queued action or a controlled failure instead of treating every unsuccessful workflow as an LLM outage.

通过共享工作流调用链区分模型、路由和工具故障。记录模型尝试、所选能力、供应商原生请求 ID、超时阶段、验证结果和恢复动作。这样运营人员可以判断用户得到的是降级但有效的回答、已排队动作，还是受控失败，而不会把所有未完成工作流都误判为 LLM 宕机。

FAQ

Official sources and further reading官方资料与延伸阅读