LLM Fallback
Fail Over Without Semantic DriftLLM 故障切换：切换路径不能带来语义漂移
Fallback is a controlled second path after a classified failure. Reliable systems bound every attempt, preserve required capabilities and record why each alternate was selected.
故障切换是分类故障后的受控第二路径。可靠系统会限制每次尝试、保留必需能力，并记录选择每个备用项的原因。
TL;DR
Timeouts, 429s, 5xx, safety blocks and capability gaps need different responses.
An available endpoint is useless if it cannot preserve required tools, context, region or policy.
Limit attempts, elapsed time, concurrency and total spend across the whole request.
Record failure class, route reason, attempt IDs, transformations, result and final disposition.
超时、429、5xx、安全阻断与能力缺口需要不同响应。
若不能保留所需工具、上下文、区域或策略，可用端点也无意义。
对整个请求限制尝试次数、耗时、并发与总成本。
记录故障类别、路由原因、尝试 ID、转换、结果与最终处置。
Fallback is policy, not a retry loop故障切换是策略，不是重试循环
A retry repeats an operation, usually against the same logical route. Fallback changes region, deployment, provider or model according to an ordered policy. That change can restore availability, but it can also alter output quality, tool behavior, context limits, data handling, price, latency and even the way usage or errors are reported.
Retry 通常在同一逻辑路由上重复操作；Fallback 则按有序策略切换区域、Deployment、供应商或模型。切换可以恢复可用性，也可能改变输出质量、工具行为、上下文限制、数据处理、价格、延迟，甚至改变 Usage 或错误的报告方式。
Define which failures permit fallback. Network timeouts, transient 5xx responses and bounded capacity errors may qualify; invalid input, authentication, policy rejection or a deterministic schema failure generally should not. A quality validator can trigger escalation, but only if its decision is reproducible and the extra latency fits the workload deadline.
需要定义哪些故障允许 Fallback。网络超时、暂时性 5xx 与有界容量错误可能符合条件；无效输入、认证、策略拒绝或确定性 Schema 失败通常不应切换。质量验证器也可以触发升级，但前提是其判断可复现，而且额外延迟仍处于工作负载截止时间内。
Start with hard eligibility: required modality and tools, output schema, policy, tenant permission, region, data-retention rule, remaining deadline and budget. Then rank only eligible alternates. Stop when the request budget is exhausted, cancellation arrives or no candidate can preserve the workload contract; a controlled error is safer than a semantically wrong success.
先执行硬性合格过滤：必需模态与工具、输出 Schema、策略、租户权限、区域、数据保留规则、剩余截止时间与预算；再只对合格备用项排序。请求预算耗尽、收到取消或没有候选能保留工作负载契约时应停止；受控错误比语义错误的“成功”更安全。
Failure classes and safe actions故障类别与安全动作
| Failure class故障类别 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Connection or timeout连接或超时 | Retry an idempotent attempt within the deadline.在截止时间内重试幂等尝试。 | Backoff, cancellation, duplicate execution and total elapsed-time budget.验证退避、取消、重复执行与总耗时预算。 |
| Rate limit or quota限流或配额 | Use an eligible alternate key, deployment, region or provider.使用合格的备用密钥、Deployment、区域或供应商。 | Honor retry hints; avoid spreading overload and crossing tenant budgets.遵循重试提示，避免扩散过载或跨越租户预算。 |
| Provider 5xx or outage供应商 5xx 或宕机 | Fail over through an explicit ordered pool.通过显式有序池执行故障转移。 | Circuit-break unhealthy endpoints and probe recovery separately.对不健康端点熔断，并独立探测恢复。 |
| Capability mismatch能力不匹配 | Reject before execution or choose a capability-equivalent candidate.执行前拒绝，或选择能力等价候选。 | Never silently drop tools, schema, modality or context requirements.绝不静默丢弃工具、结构定义、模态或上下文要求。 |
| Policy or safety block策略或安全阻断 | Return the defined policy outcome.返回已定义的策略结果。 | Do not route around a control by switching providers.不得通过切换供应商绕过控制。 |
Design fallback as a bounded state machine把故障切换设计为有界状态机
Normalize native errors without hiding status, IDs or retry hints.
Filter by capability, version, policy, region, tenant and budget before scoring.
Cap retries, alternates, elapsed time, tokens, spend and duplicate side effects.
Return success with route evidence or one controlled final error with the attempt chain.
标准化原生错误，同时保留状态、ID 与重试提示。
评分前按能力、版本、策略、区域、租户与预算过滤。
限制重试、备用项、耗时、Token、成本与重复副作用。
返回带路由证据的成功结果，或携带尝试链的单一受控最终错误。
Test failover before an outage宕机前测试故障转移
Exercise fallback through the complete application, SDK and gateway chain with representative prompts and validators. A route can return HTTP 200 yet fail the workload because JSON is invalid, a required tool was omitted or the answer no longer meets the quality floor.
应通过完整应用、SDK 与 Gateway 链路演练 Fallback，并使用代表性 Prompt 和验证器。路由即使返回 HTTP 200，也可能因为 JSON 无效、遗漏必需工具或答案低于质量下限而未完成工作负载。
- Inject connection, timeout, 429, 5xx, partial stream, invalid output and capability failures.
- Verify every alternate against modality, tool, schema, context, region, retention and policy invariants.
- Measure accepted-outcome rate, end-to-end attempts, elapsed time, token use, cost and duplicate side effects.
- Confirm one retry owner, circuit-breaker recovery and protection from synchronized retry storms.
- Test the case where fallback starts after partial output or a completed tool action.
- Exercise controlled failure and rollback when every candidate is unavailable or ineligible.
- 注入连接、超时、429、5xx、部分流、无效输出与能力故障。
- 针对模态、工具、Schema、上下文、区域、保留和策略不变量验证每个备用项。
- 测量可接受结果率、端到端尝试、耗时、Token 用量、成本与重复副作用。
- 确认唯一重试负责人、熔断恢复和同步重试风暴防护。
- 测试已经输出部分内容或完成工具动作后才启动 Fallback 的情况。
- 在所有候选不可用或不合格时演练受控失败与回滚。
Separate classification, selection and execution分离分类、选择与执行
The gateway first classifies the provider-native failure and decides whether fallback is allowed. It checks the remaining parent deadline and attempt budget, filters a versioned candidate pool by hard constraints and selects the next route. Store rejected candidates and reasons so an operator can explain why a route was not used.
Gateway 先分类供应商原生故障并判断是否允许 Fallback，再检查父请求剩余截止时间与尝试预算，按硬约束过滤版本化候选池并选择下一条路由。应保存被淘汰候选及原因，让运维人员能够解释某条路由为什么未被采用。
Each execution carries a unique attempt ID under one parent trace and consumes the same global budget. The executor cancels superseded work where possible, validates the result and either returns verified output or re-enters selection with the failed route excluded. The final controlled error includes the failure class and complete attempt chain without exposing secrets.
每次执行都在同一父 Trace 下携带唯一 Attempt ID，并消耗同一全局预算。Executor 尽可能取消被替代工作，验证结果；成功则返回已验证输出，失败则排除该路由后重新进入选择。最终受控错误应包含故障类别与完整尝试链，但不能暴露 Secret。
Production rule: availability never justifies bypassing capability, policy, tenant or safety invariants.
生产规则：可用性绝不能成为绕过能力、策略、租户或安全不变量的理由。
Fallback inference and fallback capabilities are separate推理故障切换与能力故障切换是两件事
A model gateway can fail over inference endpoints. QVeris complements that layer by discovering and calling external APIs, tools, services and live data. Treat capability failure separately: a healthy alternate model cannot repair an unavailable market-data API or transaction service unless the workflow has its own verified capability fallback.
Model Gateway 可以对推理端点执行故障切换；QVeris 补充外部 API、工具、服务与实时数据的发现和调用。能力故障应独立处理：除非工作流拥有自己的已验证能力备用项，否则健康备用模型无法修复不可用的市场数据 API 或交易服务。
Keep separate candidate pools, health state, credentials, retry budgets and idempotency keys, then connect both attempt chains with a parent trace. If the tool action completed before inference failed, resume with its verified result instead of repeating the side effect. If an alternate data source is used, validate schema, licensing, freshness and semantics before treating it as equivalent.
应分别维护候选池、健康状态、凭证、重试预算与幂等密钥，再通过父 Trace 关联两条尝试链。如果推理失败前工具动作已经完成，应携带已验证结果继续，而不是重复副作用；使用备用数据源时，也必须验证 Schema、许可、新鲜度与语义后才能视为等价。
FAQ
Only classified, policy-approved failures such as bounded timeouts, eligible rate limits or endpoint outages.
Yes, if the alternate preserves the workload's required capabilities and accepted quality envelope.
No. A safety or policy decision should return its defined outcome, not be routed around.
仅限已分类且策略批准的故障，如有界超时、可处理限流或端点宕机。
可以，前提是备用项保留负载所需能力与可接受质量范围。
不应。安全或策略决策应返回其定义结果，不得被绕过。