LLM High Availability
Design Failure Domains, Prove Recovery
LLM 高可用：设计故障域，并用演练证明可恢复
High availability is not a long fallback list. It is a tested system of independent failure domains, equivalent capabilities, bounded recovery and clear final outcomes.
高可用不是一长串故障切换。它是由独立故障域、等价能力、有界恢复与明确最终结果组成并经过测试的系统。
TL;DR
Separate provider, model, gateway, region, network, control plane and configuration risk.
Filter by tools, context, output, safety, region and data policy before failover.
Cap queue age, attempts, elapsed time, tokens, spend and duplicate side effects.
Inject failures, observe recovery, canary policy changes and retain rollback.
分离供应商、模型、网关、区域、网络、控制面与配置风险。
故障切换前按工具、上下文、输出、安全、区域与数据策略过滤。
限制队列年龄、尝试次数、耗时、Token、支出与重复副作用。
注入故障、观察恢复、灰度策略变更并保留回滚。
Availability starts with workload contracts 高可用始于工作负载契约
For each workload, define the required model capabilities, maximum latency, data region, policy, output validation and acceptable degraded modes. An endpoint belongs in the failover pool only after it passes that contract.
为每个负载定义所需模型能力、最大延迟、数据区域、策略、输出验证与可接受降级模式。端点只有通过该契约后才能进入故障切换 Pool。
List correlated risks. Two provider labels may share a cloud region, network, gateway, identity system or configuration source. Redundancy counts only when the relevant failure domains are independent.
列出相关风险。两个供应商标签可能共享 Cloud Region、Network、网关、Identity System 或配置来源。只有相关故障域独立时，冗余才有效。
Availability pattern decisions 高可用模式决策
| Pattern 模式 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Multi-zone gateway 多可用区网关 | Gateway process or zone failure is the main risk. 网关进程或可用区故障是主要风险。 | State replication, idempotency, health checks and configuration consistency. 核对状态复制、幂等、健康检查与配置一致性。 |
| Multi-provider pool 多供应商资源池 | Provider or model service failure must be tolerated. 必须容忍供应商或模型服务故障。 | Capability equivalence, data terms, quotas, semantics and quality. 核对能力等价、数据条款、配额、语义与质量。 |
| Multi-region active-active 多区域双活 | Regional loss and low recovery time justify complexity. 区域故障与低恢复时间要求值得额外复杂度。 | Residency, global state, routing, split brain, cost and failback. 核对驻留、全局状态、路由、脑裂、成本与恢复切回。 |
| Controlled degradation 受控降级 | No equivalent route exists for part of the workload. 部分负载没有等价路由。 | Explicit feature reduction, user notice, safety and recovery trigger. 核对显式功能缩减、用户提示、安全与恢复触发。 |
High-availability controls 高可用控制
Use request-path evidence and timestamped probes; avoid stale binary health.
Define trip, cooldown, half-open probes, gradual re-entry and failback.
One layer owns retries and shares a parent deadline across attempts.
Store candidate set, exclusions, route decision, attempts and final disposition.
使用请求路径证据与带时间戳探测，避免过期二元状态。
定义触发、冷却、半开探测、逐步回归与恢复切回。
一层负责重试，并在各次尝试间共享父截止时间。
保存候选集、排除原因、路由决策、尝试与最终处置。
Run recovery game days 运行恢复演练
- Fail one gateway, provider, model, region and configuration dependency at a time.
- Inject correlated failures, slow responses, broken streams and stale health.
- Verify deadlines, idempotency, tenant fairness, data policy and controlled errors.
- Exercise failback and rollback so recovery does not create a second incident.
- 依次让一个网关、供应商、模型、Region 与配置依赖失效。
- 注入相关故障、慢响应、断流与过期健康状态。
- 验证截止时间、幂等、租户公平、数据策略与受控错误。
- 演练恢复切回与回滚，避免恢复制造第二次事故。
Separate control and data planes 分离控制面与数据面
The control plane manages versioned catalogs, policies, health and rollout. The data plane validates each request, filters by capability and residency, selects a healthy route and executes a bounded attempt. Both planes are replicated independently and emit joined evidence.
控制面管理版本化目录、策略、健康与发布；数据面验证每个请求，按能力与驻留过滤，选择健康路由并执行有界尝试；两个平面独立复制并生成可关联证据。
Production rule: never fail over to an endpoint that has not passed the workload's capability and policy contract.
生产规则：绝不能故障切换到未通过该负载能力与策略契约的端点。
Design tool availability as a separate system 把工具可用性设计为独立系统
Model availability restores inference. QVeris governs external APIs, tools, services and live data. Give those capabilities separate pools, timeouts, idempotency and recovery tests, then connect both through the workflow trace.
模型可用性恢复推理；QVeris 治理外部 API、工具、服务与实时数据。为这些能力配置独立资源池、超时、幂等与恢复测试，再通过工作流调用链连接两者。
Turn LLM High Availability into an SLO and Runbook把 LLM 高可用转成 SLO 与运行手册
A multi-provider diagram is not an availability guarantee. Define which user journeys must survive, what degraded service is acceptable, how quickly the system must recover, and which evidence proves that failover preserved meaning rather than merely returning an HTTP 200.
多供应商架构图并不等于可用性保证。团队需要定义哪些用户路径必须存活、允许怎样的降级、系统应多快恢复，以及什么证据能够证明故障切换保留了业务语义，而不只是返回 HTTP 200。
| Runbook element运行手册要素 | Define before launch上线前定义 | Acceptance evidence验收证据 |
|---|---|---|
| Service objective服务目标 | Availability and latency targets by workflow, error-budget window, business hours, regional scope, and the minimum response quality that counts as success按工作流定义可用性与延迟目标、错误预算窗口、业务时段、区域范围，以及可被视为成功的最低响应质量 | A dashboard that separates transport success, valid model output, policy compliance, and end-to-end task completion仪表盘分别展示传输成功、模型输出有效、策略合规与端到端任务完成 |
| Failure domain故障域 | Provider, model family, credential, gateway, region, network, quota, policy store, and downstream tool dependencies that could fail together可能共同失效的供应商、模型族、凭证、网关、区域、网络、配额、策略存储和下游工具依赖 | Dependency maps and game-day scenarios that remove one domain at a time and expose correlated assumptions依赖地图与演练场景逐个移除故障域，并暴露隐藏的相关性假设 |
| Failover contract故障切换契约 | Eligible substitutes, capability parity, context limits, structured-output differences, retry limits, idempotency, and when to return a partial result instead合格替代项、能力等价性、上下文限制、结构化输出差异、重试上限、幂等性，以及何时应返回部分结果 | Recorded prompts, routes, transformations, outputs, and semantic checks showing the fallback still satisfies the user intent记录提示词、路由、转换、输出和语义检查，证明降级路径仍满足用户意图 |
| Recovery and ownership恢复与责任 | Detection threshold, automated and manual actions, incident owner, communication path, rollback, return-to-primary criteria, RTO, and data or conversation-state RPO检测阈值、自动与人工动作、事故负责人、沟通路径、回滚、恢复主路径标准、RTO，以及数据或会话状态 RPO | Timed drills, operator notes, alert-to-action latency, safe restoration, and follow-up changes tied to failed assumptions计时演练、操作记录、从告警到动作的延迟、安全恢复，以及针对失败假设的后续改进 |
FAQ
Enough independent routes to meet the risk objective; more untested routes add failure modes.
No. Choose architecture from business impact, residency and recovery objectives.
The minimum real capability path, not only DNS or a generic status endpoint.
足以满足风险目标的独立路由即可；更多未测试路由只会增加故障模式。
不需要。根据业务影响、驻留与恢复目标选择架构。
测试最小真实能力路径，而不只是 DNS 或通用状态端点。