LLM Fallback Routing Setup: Bound Attempts, Preserve Semantics LLM 故障切换路由设置：限制尝试次数，保留语义
Reliable fallback starts with a workload contract and ends with fault tests. Do not add a generic retry loop; define which failures, alternates, budgets and semantic changes are acceptable.
可靠故障切换从工作负载契约开始，以故障测试结束。不要添加通用重试循环；应定义可接受的故障、备用项、预算与语义变化。
TL;DR
List required model features, tools, context, output, policy and deadline.
Separate retryable transport faults from capability, policy and input failures.
Cap attempts, elapsed time, tokens, spend and duplicate side effects.
Test every route under timeouts, 429s, 5xx, broken streams and no-candidate states.
列出所需模型功能、工具、上下文、输出、策略与截止时间。
区分可重试传输故障与能力、策略及输入故障。
限制次数、耗时、Token、成本与重复副作用。
在超时、429、5xx、异常流与无候选状态下测试每条路由。
A safe setup sequence 安全设置顺序
Start with one workload and one verified alternate. Create a failure taxonomy, an eligibility pool and a total request budget before choosing retry order.
从一个工作负载与一个已验证备用项开始。在选择重试顺序前，建立故障分类、合格池与总请求预算。
Attach one attempt ID to every execution under a parent trace. Record the failure, exclusion rules, selected alternate, transformations and final disposition.
每次执行在父调用链下附加唯一 Attempt ID，并记录故障、排除规则、所选备用项、转换与最终处置。
Fallback configuration decisions 故障切换配置决策
| Decision 决策 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Trigger classes 触发类别 | Timeout, connection, approved 429 or transient 5xx. 超时、连接、已批准 429 或瞬态 5xx。 | Do not fallback around policy blocks, invalid input or missing required capabilities. 不得绕过策略阻断、无效输入或缺失必需能力。 |
| Eligible pool 合格池 | Alternates passing tools, context, region, safety and quality contracts. 通过工具、上下文、区域、安全与质量契约的备用项。 | Version the catalog and reject stale or unknown capability data. 版本化目录，并拒绝过期或未知能力数据。 |
| Budgets 预算 | Per-attempt timeout plus total time, attempts, token and spend caps. 单次超时加总时间、次数、Token 与成本上限。 | Count retries across SDK, gateway and provider layers once. 跨 SDK、网关与供应商层统一计算重试。 |
| Final outcome 最终结果 | Verified response or one controlled error with the attempt chain. 已验证响应，或带尝试链的单一受控错误。 | Never return an unexplained substitute or partial success. 绝不返回无法解释的替代结果或部分成功。 |
Configuration checklist 配置检查清单
Normalize errors while preserving native code, ID and retry hints.
Compare tools, schema, context, system instructions and output validators.
Protect tool calls and external actions from duplicate execution.
Alert on fallback rate, exhausted budgets, route drift and quality regression.
标准化错误，同时保留原生 Code、ID 与重试提示。
比较工具、结构定义、上下文、System Instruction 与输出验证器。
保护工具调用与外部动作免于重复执行。
对故障切换率、预算耗尽、路由漂移与质量回归告警。
Fault-test before launch 上线前故障测试
- Inject each approved trigger and verify the exact alternate order.
- Confirm hard constraints remain true after every provider or model switch.
- Test exhausted budgets, cancellation and controlled final errors.
- Canary one workload and keep a one-switch rollback.
- 注入每个批准触发器，并验证准确备用顺序。
- 确认每次供应商或模型切换后硬约束仍成立。
- 测试预算耗尽、取消与受控最终错误。
- 灰度一个工作负载，并保留一键回滚。
Use a bounded fallback state machine 使用有界故障切换状态机
The gateway classifies the failure, checks the remaining request budget, filters the versioned pool, selects the next eligible route and executes one attempt. The state machine stops on verified success, budget exhaustion or no eligible candidate.
网关分类故障、检查剩余请求预算、过滤版本化资源池、选择下一个合格路由并执行一次尝试。状态机在已验证成功、预算耗尽或无合格候选时停止。
Production rule: every fallback must preserve hard workload and policy constraints.
生产规则：每次故障切换都必须保留工作负载与策略硬约束。
Add tool fallback as a separate workflow concern 把工具故障切换作为独立工作流关注点
Model fallback restores inference. QVeris handles discovery and execution of external APIs, tools, services and live data. Give those capabilities their own availability and idempotency policy, then connect both attempt chains through the workflow trace.
模型故障切换恢复推理；QVeris 处理外部 API、工具、服务与实时数据的发现和执行。为这些能力定义独立可用性与幂等策略，再通过工作流调用链连接两条尝试链。
Encode fallback policy as data 把故障切换 Policy 编码为数据
Keep route order, eligibility, timeout, and attempt budget in versioned configuration. The application supplies workload requirements; the router filters candidates before it considers price or latency.
把路由 Order、Eligibility、Timeout 与 Attempt Budget 保存在版本化配置中。应用提供工作负载要求；路由器应先过滤不合格候选，再考虑价格或延迟。
version: 7
workloads:
support-answer:
requires: [json_schema, tool_calls]
max_attempts: 2
deadline_ms: 12000
routes:
- alias: quality-primary
timeout_ms: 8000
- alias: quality-backup
timeout_ms: 3500
only_on: [timeout, 429, 502, 503, 504]
never_retry: [401, 403, 404, invalid_request]
duplicate_protection: required
- Validate every alias against the required capability set at deployment time.
- Simulate primary timeout, 429, malformed response, and partial stream before rollout.
- Include policy version and attempted routes in the final trace.
- 部署时根据所需能力集合验证每个别名。
- 发布前模拟 Primary Timeout、429、Malformed Response 与 Partial Stream。
- 在最终调用链中包含 Policy Version 与已尝试路由。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
The Production Operating Model for LLM fallback routingLLM 故障切换路由的生产运营模型
For Set Up LLM Fallback Routing, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“设置 LLM 故障切换 Routing”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory policy order, retryable versus terminal errors, route eligibility, cooldown, circuit state, semantic checks, budget caps, hedging, attempt history, and operator override. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点策略顺序、可重试与终止错误、路由资格、冷却、熔断状态、语义检查、预算上限、Hedging、尝试历史和人工覆盖。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For Set Up LLM Fallback Routing, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“设置 LLM 故障切换 Routing”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate Set Up LLM Fallback Routing, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“设置 LLM 故障切换 Routing”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating Set Up LLM Fallback Routing, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“设置 LLM 故障切换 Routing”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
Start with the smallest verified pool that meets the availability objective.
No. Honor retry hints, budgets and policy; queue or backoff may be safer.
Give one execution layer ownership to prevent nested amplification.
从满足可用性目标的最小已验证资源池开始。
不应。遵循重试提示、预算与策略；排队或退避可能更安全。
由一个执行层负责，防止嵌套放大。