LLM Fallback Routing Setup: Bound Attempts, Preserve SemanticsLLM 故障切换路由设置：限制尝试次数，保留语义
Reliable fallback starts with a workload contract and ends with fault tests. Do not add a generic retry loop; define which failures, alternates, budgets and semantic changes are acceptable.
可靠故障切换从工作负载契约开始，以故障测试结束。不要添加通用重试循环；应定义可接受的故障、备用项、预算与语义变化。
TL;DR
List required model features, tools, context, output, policy and deadline.
Separate retryable transport faults from capability, policy and input failures.
Cap attempts, elapsed time, tokens, spend and duplicate side effects.
Test every route under timeouts, 429s, 5xx, broken streams and no-candidate states.
列出所需模型功能、工具、上下文、输出、策略与截止时间。
区分可重试传输故障与能力、策略及输入故障。
限制次数、耗时、Token、成本与重复副作用。
在超时、429、5xx、异常流与无候选状态下测试每条路由。
A safe setup sequence安全设置顺序
Start with one clearly defined workload and one verified alternate rather than a long provider chain. Write the workload contract first: required modality, context size, tool schema, structured-output rules, region, safety policy, maximum latency and acceptable quality floor. The alternate enters the pool only after it passes those same gates.
应从一个定义清晰的工作负载和一个已验证备用项开始，而不是先堆出很长的供应商链。首先写明工作负载契约：必需模态、上下文大小、工具 Schema、结构化输出规则、区域、安全策略、最大延迟与最低质量标准。备用项只有通过同一组门禁后才能进入资源池。
Next create a failure taxonomy. Transport timeouts, transient 5xx responses and bounded capacity errors may justify another attempt; authentication, invalid input, policy rejection and deterministic schema failures usually do not. Define a total request deadline, attempt limit and spend budget before choosing the order, so fallback cannot turn one failed call into an unbounded sequence.
随后建立故障分类。网络超时、暂时性 5xx 和有界容量错误可能适合再次尝试；认证失败、无效输入、策略拒绝和确定性 Schema 错误通常不应重试。在确定顺序前先设置总请求截止时间、尝试次数与成本预算，防止一次失败调用演变成无限序列。
Attach one attempt ID to every execution under a parent trace. Record the failure class, exclusion rules, candidate pool version, selected alternate, any input or output transformation and final disposition. This evidence is what lets an operator explain why fallback happened and reproduce the same routing decision.
每次执行都应在父 Trace 下附加唯一 Attempt ID，并记录故障类别、排除规则、候选池版本、所选备用项、输入或输出转换以及最终处置。这些证据让运维人员能够解释为什么发生故障切换，并重现当时的路由决策。
Fallback configuration decisions故障切换配置决策
| Decision决策 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Trigger classes触发类别 | Timeout, connection, approved 429 or transient 5xx.超时、连接、已批准 429 或瞬态 5xx。 | Do not fallback around policy blocks, invalid input or missing required capabilities.不得绕过策略阻断、无效输入或缺失必需能力。 |
| Eligible pool合格池 | Alternates passing tools, context, region, safety and quality contracts.通过工具、上下文、区域、安全与质量契约的备用项。 | Version the catalog and reject stale or unknown capability data.版本化目录，并拒绝过期或未知能力数据。 |
| Budgets预算 | Per-attempt timeout plus total time, attempts, token and spend caps.单次超时加总时间、次数、Token 与成本上限。 | Count retries across SDK, gateway and provider layers once.跨 SDK、网关与供应商层统一计算重试。 |
| Final outcome最终结果 | Verified response or one controlled error with the attempt chain.已验证响应，或带尝试链的单一受控错误。 | Never return an unexplained substitute or partial success.绝不返回无法解释的替代结果或部分成功。 |
Configuration checklist配置检查清单
Normalize errors while preserving native code, ID and retry hints.
Compare tools, schema, context, system instructions and output validators.
Protect tool calls and external actions from duplicate execution.
Alert on fallback rate, exhausted budgets, route drift and quality regression.
标准化错误，同时保留原生 Code、ID 与重试提示。
比较工具、结构定义、上下文、System Instruction 与输出验证器。
保护工具调用与外部动作免于重复执行。
对故障切换率、预算耗尽、路由漂移与质量回归告警。
Fault-test before launch上线前故障测试
Run the tests through the real SDK, gateway and worker stack. A unit test of the route selector will not reveal duplicated SDK retries, partial streaming responses or a cancelled client whose background attempt keeps running. Use representative prompts and validators so the exercise checks answer behavior as well as availability.
测试必须穿过真实的 SDK、网关和 Worker 链路。只对路由选择器做单元测试，无法发现 SDK 重复重试、部分流式响应，或客户端取消后后台尝试仍在运行等问题。使用代表性 Prompt 与验证器，让演练同时检查答案行为和可用性。
- Inject each approved trigger and verify the exact alternate order and stop conditions.
- Confirm hard capability, policy, region and data-retention constraints remain true after every switch.
- Test exhausted deadlines, attempt and spend budgets, cancellation propagation and controlled final errors.
- Compare output validity, tool behavior, tail latency and total cost across primary and fallback paths.
- Canary one workload, monitor it by route and keep a one-switch rollback to the previous policy version.
- 注入每个批准触发器，并验证准确的备用顺序与停止条件。
- 确认每次切换后，能力、策略、区域与数据保留等硬约束仍然成立。
- 测试截止时间、尝试次数和成本预算耗尽，取消传播以及受控最终错误。
- 比较主路径与备用路径的输出有效性、工具行为、尾延迟和总成本。
- 先灰度一个工作负载，按路由监控，并保留一键恢复旧策略版本的能力。
Use a bounded fallback state machine使用有界故障切换状态机
The gateway classifies the failure, checks the remaining parent deadline and attempt budget, filters the versioned pool and selects the next eligible route. It executes one attempt, validates the result and updates the state. A successful HTTP response is not sufficient when the workload requires valid JSON, a particular tool call or evidence-backed output.
网关先分类故障，检查父请求剩余截止时间与尝试预算，再过滤版本化资源池并选择下一个合格路由。执行一次尝试后还要验证结果并更新状态。对于要求有效 JSON、特定工具调用或带证据输出的工作负载，HTTP 成功并不等于任务成功。
The state machine stops on verified success, budget exhaustion, cancellation or no eligible candidate. It must also remember which routes have already failed so a loop cannot bounce between two endpoints. Keep retry delay inside the parent deadline and expose the final failure class plus attempted routes to the caller.
状态机在已验证成功、预算耗尽、收到取消或没有合格候选项时停止。它还必须记住已经失败的路由，避免在两个端点之间来回循环。所有重试等待都要计入父截止时间，并向调用方返回最终故障类别与已尝试路由。
Production rule: every fallback must preserve hard workload and policy constraints; availability never overrides eligibility.
生产规则：每次故障切换都必须保留工作负载与策略硬约束；可用性不能凌驾于合格性之上。
Add tool fallback as a separate workflow concern把工具故障切换作为独立工作流关注点
Model fallback restores inference. QVeris handles discovery and execution of external APIs, tools, services and live data. Give those capabilities their own eligibility pool, freshness requirement, availability policy and retry budget. An alternate model is not automatically entitled to a tool, and an alternate data source is not automatically semantically equivalent.
模型故障切换用于恢复推理，QVeris 则负责发现并执行外部 API、工具、服务与实时数据。应为这些能力单独定义合格池、新鲜度要求、可用性策略与重试预算。备用模型并不会自动获得某项工具权限，备用数据源也不会天然具备相同语义。
Connect the two attempt chains through the workflow trace while preserving separate action IDs and idempotency keys. If a tool action completed before the model failed, resume with the verified result rather than repeating the side effect. If the result is time-sensitive, refresh it explicitly and record why a second call was necessary.
通过工作流 Trace 关联两条尝试链，同时保留独立的 Action ID 与幂等密钥。如果模型失败前工具动作已经完成，应携带已验证结果继续，而不是重复副作用；如果结果具有时效性，则明确刷新并记录为什么需要第二次调用。
Encode fallback policy as data把故障切换 Policy 编码为数据
Keep route order, eligibility, timeout, and attempt budget in versioned configuration. The application supplies workload requirements; the router filters candidates before it considers price or latency.
把路由 Order、Eligibility、Timeout 与 Attempt Budget 保存在版本化配置中。应用提供工作负载要求；路由器应先过滤不合格候选，再考虑价格或延迟。
version: 7
workloads:
support-answer:
requires: [json_schema, tool_calls]
max_attempts: 2
deadline_ms: 12000
routes:
- alias: quality-primary
timeout_ms: 8000
- alias: quality-backup
timeout_ms: 3500
only_on: [timeout, 429, 502, 503, 504]
never_retry: [401, 403, 404, invalid_request]
duplicate_protection: required- Validate every alias against the required capability set at deployment time.
- Simulate primary timeout, 429, malformed response, and partial stream before rollout.
- Include policy version and attempted routes in the final trace.
- 部署时根据所需能力集合验证每个别名。
- 发布前模拟 Primary Timeout、429、Malformed Response 与 Partial Stream。
- 在最终调用链中包含 Policy Version 与已尝试路由。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
FAQ
Start with the smallest verified pool that meets the availability objective.
No. Honor retry hints, budgets and policy; queue or backoff may be safer.
Give one execution layer ownership to prevent nested amplification.
从满足可用性目标的最小已验证资源池开始。
不应。遵循重试提示、预算与策略；排队或退避可能更安全。
由一个执行层负责，防止嵌套放大。