LLM Fallback
Fail Over Without Semantic Drift
LLM 故障切换：切换路径不能带来语义漂移
Fallback is a controlled second path after a classified failure. Reliable systems bound every attempt, preserve required capabilities and record why each alternate was selected.
故障切换是分类故障后的受控第二路径。可靠系统会限制每次尝试、保留必需能力，并记录选择每个备用项的原因。
TL;DR
Timeouts, 429s, 5xx, safety blocks and capability gaps need different responses.
An available endpoint is useless if it cannot preserve required tools, context, region or policy.
Limit attempts, elapsed time, concurrency and total spend across the whole request.
Record failure class, route reason, attempt IDs, transformations, result and final disposition.
超时、429、5xx、安全阻断与能力缺口需要不同响应。
若不能保留所需工具、上下文、区域或策略，可用端点也无意义。
对整个请求限制尝试次数、耗时、并发与总成本。
记录故障类别、路由原因、尝试 ID、转换、结果与最终处置。
Fallback is policy, not a retry loop 故障切换是策略，不是重试循环
A retry repeats an operation, often against the same endpoint. A fallback changes region, deployment, provider or model according to an ordered policy. The change can restore availability, but it can also change output quality, tool behavior, context, data handling, price or latency.
Retry 通常在同一端点重复操作；故障切换则按有序策略切换区域、Deployment、供应商或模型。切换可以恢复可用性，也可能改变输出质量、工具行为、上下文、数据处理、价格或延迟。
Start with hard eligibility: required capabilities, policy, tenant permissions, region and remaining budget. Then rank eligible alternates. Stop when the request budget is exhausted or no candidate can preserve the workload contract.
先执行硬性合格过滤：必需能力、策略、租户权限、区域与剩余预算；随后对合格备用项排序。当请求预算耗尽或没有候选能保留工作负载契约时停止。
Failure classes and safe actions 故障类别与安全动作
| Failure class 故障类别 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Connection or timeout 连接或超时 | Retry an idempotent attempt within the deadline. 在截止时间内重试幂等尝试。 | Backoff, cancellation, duplicate execution and total elapsed-time budget. 验证退避、取消、重复执行与总耗时预算。 |
| Rate limit or quota 限流或配额 | Use an eligible alternate key, deployment, region or provider. 使用合格的备用密钥、Deployment、区域或供应商。 | Honor retry hints; avoid spreading overload and crossing tenant budgets. 遵循重试提示，避免扩散过载或跨越租户预算。 |
| Provider 5xx or outage 供应商 5xx 或宕机 | Fail over through an explicit ordered pool. 通过显式有序池执行故障转移。 | Circuit-break unhealthy endpoints and probe recovery separately. 对不健康端点熔断，并独立探测恢复。 |
| Capability mismatch 能力不匹配 | Reject before execution or choose a capability-equivalent candidate. 执行前拒绝，或选择能力等价候选。 | Never silently drop tools, schema, modality or context requirements. 绝不静默丢弃工具、结构定义、模态或上下文要求。 |
| Policy or safety block 策略或安全阻断 | Return the defined policy outcome. 返回已定义的策略结果。 | Do not route around a control by switching providers. 不得通过切换供应商绕过控制。 |
Design fallback as a bounded state machine 把故障切换设计为有界状态机
Normalize native errors without hiding status, IDs or retry hints.
Filter by capability, version, policy, region, tenant and budget before scoring.
Cap retries, alternates, elapsed time, tokens, spend and duplicate side effects.
Return success with route evidence or one controlled final error with the attempt chain.
标准化原生错误，同时保留状态、ID 与重试提示。
评分前按能力、版本、策略、区域、租户与预算过滤。
限制重试、备用项、耗时、Token、成本与重复副作用。
返回带路由证据的成功结果，或携带尝试链的单一受控最终错误。
Test failover before an outage 宕机前测试故障转移
- Inject connection, timeout, 429, 5xx, malformed stream and capability failures.
- Verify every alternate against tool, schema, context, region and policy invariants.
- Measure end-to-end attempt count, elapsed time, token use and duplicate side effects.
- Confirm circuit breaker recovery and prevent synchronized retry storms.
- Exercise controlled failure and rollback when every candidate is unavailable.
- 注入连接、超时、429、5xx、异常流与能力故障。
- 针对工具、结构定义、上下文、区域与策略不变量验证每个备用项。
- 测量端到端尝试次数、耗时、Token 用量与重复副作用。
- 确认熔断恢复，并阻止同步重试风暴。
- 在所有候选不可用时演练受控失败与回滚。
Separate classification, selection and execution 分离分类、选择与执行
The gateway classifies a native failure, checks the remaining request budget, filters a versioned candidate pool and selects the next endpoint. Each attempt carries a unique ID under one parent trace. The executor cancels superseded work where possible and returns either a verified response or a controlled error with the complete chain.
网关分类原生故障、检查剩余请求预算、过滤版本化候选池并选择下一个端点。每次尝试在同一父调用链下拥有唯一 ID；执行器尽可能取消已被替代的工作，并返回已验证响应或包含完整链路的受控错误。
Production rule: availability never justifies bypassing capability, policy or safety invariants.
生产规则：可用性绝不能成为绕过能力、策略或安全不变量的理由。
Fallback inference and fallback capabilities are separate 推理故障切换与能力故障切换是两件事
A model gateway can fail over inference endpoints. QVeris complements that layer by discovering and calling external APIs, tools, services and live data. Treat tool failure separately: a healthy alternate model cannot repair an unavailable external capability unless the workflow has its own verified fallback.
模型网关可以对推理端点执行故障转移；QVeris 补充外部 API、工具、服务与实时数据的发现和调用。工具故障应独立处理：除非工作流拥有自己的已验证备用能力，否则健康的备用模型无法修复不可用的外部能力。
The Production Operating Model for LLM fallbackLLM 故障切换的生产运营模型
For LLM Fallback, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“LLM 故障切换”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory failure classification, eligible capability-equivalent substitutes, quality floors, context and tool parity, retry limits, state, idempotency, partial results, and return-to-primary. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点失败分类、能力等价的合格替代项、质量底线、上下文与工具等价、重试上限、状态、幂等、部分结果和恢复主路径。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For LLM Fallback, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“LLM 故障切换”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate LLM Fallback, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“LLM 故障切换”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating LLM Fallback, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“LLM 故障切换”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
Only classified, policy-approved failures such as bounded timeouts, eligible rate limits or endpoint outages.
Yes, if the alternate preserves the workload's required capabilities and accepted quality envelope.
No. A safety or policy decision should return its defined outcome, not be routed around.
仅限已分类且策略批准的故障，如有界超时、可处理限流或端点宕机。
可以，前提是备用项保留负载所需能力与可接受质量范围。
不应。安全或策略决策应返回其定义结果，不得被绕过。