OpenAI Rate Limit Fallback
Back Off Before You Route Around
OpenAI 限流后的故障切换：先有序退避，再考虑换路由
A 429 is a capacity signal, not permission to spray requests across keys or providers. Parse retry hints, respect budgets and use only policy-approved alternates.
429 是容量信号，不是把请求撒向多个密钥或供应商的许可。解析重试提示、尊重预算，并只使用策略批准的备用项。
TL;DR
Preserve status, request ID and retry or limit headers when available.
If the deadline allows, waiting can be safer than changing execution semantics.
Check model, tools, region, policy, data and budget before switching.
Coordinate SDK, gateway and workflow retries under one request budget.
可用时保留状态、请求 ID 以及重试或限制 Header。
若截止时间允许，等待可能比改变执行语义更安全。
切换前检查模型、工具、区域、策略、数据与预算。
在一个请求预算下协调 SDK、网关与工作流重试。
429 response options 429 响应选择
A rate limit may apply to a project, organization, user, model, deployment or token budget. Interpret the scope before deciding whether an alternate is actually independent.
限流可能作用于项目、组织、用户、模型、Deployment 或 Token 预算。必须先解释范围，再判断备用项是否真正独立。
Choose among fail-fast, bounded queue, jittered backoff or a verified alternate based on deadline and workload contract. Record the branch taken.
根据截止时间与工作负载契约，在快速失败、有界队列、抖动退避或已验证备用项间选择，并记录所走分支。
Rate-limit response paths 限流响应路径
| Path 路径 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Fail fast 快速失败 | Interactive workload cannot wait or substitute. 交互负载无法等待或替代。 | Return a controlled error with retry guidance and request ID. 返回带重试指导与请求 ID 的受控错误。 |
| Queue 排队 | Deadline and queue capacity are known. 截止时间与队列容量已知。 | Bound queue age, depth, cancellation and tenant fairness. 限制队列年龄、深度、取消与租户公平。 |
| Backoff 退避 | Same endpoint is expected to recover within budget. 同一端点预计在预算内恢复。 | Honor hints, add jitter and stop at the parent deadline. 遵循提示、增加抖动并在父截止时间停止。 |
| Alternate 备用项 | A separate eligible route preserves the contract. 独立合格路由保留契约。 | Do not hop quotas, bypass policy or duplicate tool actions. 不得跳配额、绕策略或重复工具动作。 |
Safeguards for every branch 每条分支的保护
Count all attempts, waiting, tokens and spend under one parent request.
Deduplicate tool calls and downstream actions before retry.
Enforce tenant and project budgets across every key and provider.
Record the 429, decision, alternate, attempt and recovery outcome.
在一个父请求下计算所有尝试、等待、Token 与成本。
重试前去重工具调用与下游动作。
跨每个密钥与供应商执行租户与项目预算。
记录 429、决策、备用项、尝试与恢复结果。
Exercise the 429 runbook 演练 429 Runbook
- Inject 429s with and without retry hints at each possible scope.
- Verify queue limits, jitter, cancellation and request deadlines.
- Confirm alternate routes preserve tools, data region and safety policy.
- Review alerts, evidence and recovery after the incident closes.
- 在每个可能范围注入带与不带重试提示的 429。
- 验证队列限制、抖动、取消与请求截止时间。
- 确认备用路由保留工具、数据区域与安全策略。
- 事故结束后复盘告警、证据与恢复。
Centralize rate-limit decisions 集中限流决策
The gateway normalizes the 429, reads native hints, checks the parent budget and selects fail-fast, queue, backoff or an eligible alternate. One retry owner prevents amplification; idempotency protects external actions.
网关标准化 429、读取原生提示、检查父预算，并选择快速失败、排队、退避或合格备用项。单一重试负责人防止放大，幂等性保护外部动作。
Production rule: never treat extra keys or providers as a way to evade an enforced quota.
生产规则：绝不能把额外密钥或供应商当作规避既有限额的方式。
Keep external actions safe during retries 重试期间保护外部动作
QVeris calls external APIs, tools, services and live data after or during inference. Propagate idempotency keys and the parent trace so a model retry cannot duplicate a real-world action.
QVeris 在推理期间或之后调用外部 API、工具、服务与实时数据。传播 Idempotency 密钥与父调用链，避免模型重试重复真实世界动作。
Fallback only after classifying the 429 只有分类 429 后才执行故障切换
A 429 can mean a short burst, account quota, per-model capacity, or policy. Respect Retry-After when present, consume a shared attempt budget, and fall back only to a model that has passed the same capability and quality gates.
429 可能表示短时突发、账户配额、单模型容量或策略限制。存在 Retry-After 时应遵守它，使用共享 Attempt Budget，并且只回退到通过相同能力与质量门槛的模型。
async def call_with_fallback(request, routes, max_attempts=2):
errors = []
for attempt, route in enumerate(routes[:max_attempts], start=1):
try:
return await route.client.call(
model=route.model,
request=request,
timeout=route.timeout,
idempotency_key=request.id,
)
except RateLimitError as error:
errors.append((route.name, "429", error.retry_after))
if error.retry_after and error.retry_after <= 2:
await asyncio.sleep(error.retry_after)
continue
raise AllRoutesLimited(errors)
- Do not retry authentication, invalid schema, or deterministic policy failures.
- Prevent both SDK and gateway from independently multiplying attempts.
- Log route, attempt, Retry-After, latency, final result, and duplicate-prevention key.
- 不要重试认证、无效结构定义或确定性策略失败。
- 防止 SDK 与网关各自独立放大重试次数。
- 记录路由、Attempt、Retry-After、延迟、最终结果与防重复密钥。
Verified implementation reference: OpenAI rate limits.
实施参考已根据官方资料核验：OpenAI rate limits。
The Production Operating Model for OpenAI rate-limit fallbackOpenAI 限流故障切换的生产运营模型
For OpenAI Rate Limit Fallback, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“OpenAI 限流故障切换”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory 429 subtype and headers, quota window, request and token budget, queue delay, retry-after, cooldown, equivalent fallback capability, duplicate prevention, and attempt lineage. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点429 子类型与响应头、配额窗口、请求与 Token 预算、队列延迟、Retry-After、冷却、等价备用能力、防重复和尝试血缘。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For OpenAI Rate Limit Fallback, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“OpenAI 限流故障切换”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate OpenAI Rate Limit Fallback, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“OpenAI 限流故障切换”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating OpenAI Rate Limit Fallback, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“OpenAI 限流故障切换”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
No. Retry only within deadline, budget and provider guidance.
Yes only to an independently eligible and policy-approved route.
Use one retry owner, global concurrency limits, jitter and a circuit breaker.
不应。仅在截止时间、预算与供应商指导内重试。
只能切换到独立合格且策略批准的路由。
使用单一重试负责人、全局并发限制、抖动与熔断。