OpenAI Rate Limit Fallback
Back Off Before You Route AroundOpenAI 限流后的故障切换：先有序退避，再考虑换路由
A 429 is a capacity signal, not permission to spray requests across keys or providers. Parse retry hints, respect budgets and use only policy-approved alternates.
429 是容量信号，不是把请求撒向多个密钥或供应商的许可。解析重试提示、尊重预算，并只使用策略批准的备用项。
TL;DR
Preserve status, request ID and retry or limit headers when available.
If the deadline allows, waiting can be safer than changing execution semantics.
Check model, tools, region, policy, data and budget before switching.
Coordinate SDK, gateway and workflow retries under one request budget.
可用时保留状态、请求 ID 以及重试或限制 Header。
若截止时间允许，等待可能比改变执行语义更安全。
切换前检查模型、工具、区域、策略、数据与预算。
在一个请求预算下协调 SDK、网关与工作流重试。
429 response options429 响应选择
A rate limit may apply to a project, organization, user, model, deployment, request rate or token budget. Interpret that scope before deciding whether another route is actually independent. A second model under the same project quota may fail for the same reason, while a different deployment can still be unsuitable if it changes the context window, tool schema or data-residency contract.
限流可能作用于项目、组织、用户、模型、Deployment、请求速率或 Token 预算。必须先判断限制范围，再确认另一条路由是否真正独立。同一项目配额下的第二个模型可能因相同原因继续失败；而不同 Deployment 即使可用，如果改变了上下文窗口、工具结构或数据驻留约定，也不能直接视为合格备用项。
Choose among fail-fast, a bounded queue, jittered backoff or a verified alternate based on the parent deadline and workload contract. Interactive requests usually favor a quick, explainable response; asynchronous jobs can wait only while queue age remains inside their service-level objective. Record the 429 scope, provider hint, selected branch and remaining budget so the decision can be explained later.
应根据父请求截止时间与工作负载契约，在快速失败、有界队列、抖动退避或已验证备用项之间选择。交互请求通常更适合快速、可解释地返回；异步任务只有在队列等待时间仍处于服务目标之内时才适合排队。记录 429 的作用范围、供应商提示、所选分支与剩余预算，才能在事后解释这次决策。
Rate-limit response paths限流响应路径
| Path路径 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Fail fast快速失败 | Interactive workload cannot wait or substitute.交互负载无法等待或替代。 | Return a controlled error with retry guidance and request ID.返回带重试指导与请求 ID 的受控错误。 |
| Queue排队 | Deadline and queue capacity are known.截止时间与队列容量已知。 | Bound queue age, depth, cancellation and tenant fairness.限制队列年龄、深度、取消与租户公平。 |
| Backoff退避 | Same endpoint is expected to recover within budget.同一端点预计在预算内恢复。 | Honor hints, add jitter and stop at the parent deadline.遵循提示、增加抖动并在父截止时间停止。 |
| Alternate备用项 | A separate eligible route preserves the contract.独立合格路由保留契约。 | Do not hop quotas, bypass policy or duplicate tool actions.不得跳配额、绕策略或重复工具动作。 |
Safeguards for every branch每条分支的保护
Count all attempts, waiting, tokens and spend under one parent request.
Deduplicate tool calls and downstream actions before retry.
Enforce tenant and project budgets across every key and provider.
Record the 429, decision, alternate, attempt and recovery outcome.
在一个父请求下计算所有尝试、等待、Token 与成本。
重试前去重工具调用与下游动作。
跨每个密钥与供应商执行租户与项目预算。
记录 429、决策、备用项、尝试与恢复结果。
Exercise the 429 runbook演练 429 Runbook
A fallback design is not proven by a successful happy-path request. Test it under realistic concurrency and with the same SDK, gateway and worker retry settings used in production. The important result is not merely “a response arrived,” but that recovery stayed within the end-to-end deadline without multiplying attempts or changing the answer contract.
正常请求成功并不能证明故障切换设计可靠。应使用与生产环境相同的 SDK、网关和 Worker 重试设置，在真实并发条件下进行演练。关键结果不只是“最终拿到了响应”，而是恢复过程没有放大尝试次数、没有改变答案契约，并且始终处于端到端截止时间内。
- Inject 429s with and without retry hints at project, model and token-budget scopes.
- Verify queue depth and age limits, jitter, cancellation propagation and request deadlines.
- Confirm alternate routes preserve required tools, structured output, data region and safety policy.
- Test simultaneous failures so a shared fallback does not become the next bottleneck.
- Review alerts, trace evidence, duplicate suppression and controlled recovery after the incident closes.
- 在项目、模型和 Token 预算等不同范围注入带与不带重试提示的 429。
- 验证队列深度与等待上限、抖动、取消传播和请求截止时间。
- 确认备用路由保留必需工具、结构化输出、数据区域与安全策略。
- 测试多条主路由同时失败，避免共享备用项成为新的瓶颈。
- 事故结束后复盘告警、Trace 证据、重复抑制与受控恢复过程。
Centralize rate-limit decisions集中限流决策
The gateway should normalize the 429, retain the provider's native error details, check the parent budget and select fail-fast, queue, backoff or an eligible alternate. Applications then submit one request contract instead of embedding different retry loops in every SDK integration. Central policy also makes tenant fairness, concurrency ceilings and circuit-breaker state consistent across services.
网关应标准化 429，同时保留供应商原始错误信息，检查父请求预算，并选择快速失败、排队、退避或合格备用项。这样应用只需提交一份请求契约，不必在每个 SDK 集成中分别嵌入不同的重试循环；租户公平性、并发上限和熔断状态也能在各服务之间保持一致。
Assign exactly one retry owner. If a provider SDK, gateway and job worker each retry twice, one user request can fan out into many billable attempts and worsen the limit. Pass a shared attempt count, absolute deadline and trace ID through every layer; when the budget is exhausted, return a controlled result rather than starting another hidden loop.
必须明确唯一的重试负责人。如果供应商 SDK、网关和任务 Worker 都各自重试两次，一次用户请求就可能扩散成大量计费调用，反而加剧限流。每一层都应传递共享 Attempt Count、绝对截止时间和 Trace ID；预算耗尽后返回受控结果，而不是再启动一轮隐藏重试。
Production rule: never treat extra keys or providers as a way to evade an enforced quota. Alternate routes are for contract-preserving resilience, not quota hopping.
生产规则：绝不能把额外密钥或供应商当作规避既有限额的方式。备用路由用于在保持契约的前提下提高韧性，而不是跳配额。
Keep external actions safe during retries重试期间保护外部动作
QVeris can call external APIs, tools, services and live data during an agent workflow. A model request may be safe to repeat, while the tool action it already triggered—placing an order, sending a message or updating a record—may not be. Keep the inference attempt and the external action as separate states rather than replaying the entire workflow after every 429.
QVeris 可在 Agent 工作流中调用外部 API、工具、服务与实时数据。模型请求本身可能可以重复，但它已经触发的工具动作——例如下单、发消息或更新记录——未必可以重放。因此，应把推理尝试和外部动作作为两个独立状态管理，而不是每次遇到 429 都重放整个工作流。
Propagate the parent trace, action ID and idempotency key into the tool layer, then persist whether the action was planned, submitted, confirmed or rejected. On recovery, resume from the last verified state and re-read time-sensitive data when freshness matters. This prevents a rate-limit fallback from silently duplicating a real-world side effect or acting on stale evidence.
将父 Trace、Action ID 与幂等密钥传播到工具层，并持久化动作处于已规划、已提交、已确认还是已拒绝状态。恢复时从最后一个已验证状态继续；如果数据时效性重要，还应重新读取实时数据。这样可避免限流故障切换悄悄重复真实世界副作用，或基于过期证据继续执行。
Fallback only after classifying the 429只有分类 429 后才执行故障切换
A 429 can mean a short burst, account quota, per-model capacity, or policy. Respect Retry-After when present, consume a shared attempt budget, and fall back only to a model that has passed the same capability and quality gates.
429 可能表示短时突发、账户配额、单模型容量或策略限制。存在 Retry-After 时应遵守它，使用共享 Attempt Budget，并且只回退到通过相同能力与质量门槛的模型。
async def call_with_fallback(request, routes, max_attempts=2):
errors = []
for attempt, route in enumerate(routes[:max_attempts], start=1):
try:
return await route.client.call(
model=route.model,
request=request,
timeout=route.timeout,
idempotency_key=request.id,
)
except RateLimitError as error:
errors.append((route.name, "429", error.retry_after))
if error.retry_after and error.retry_after <= 2:
await asyncio.sleep(error.retry_after)
continue
raise AllRoutesLimited(errors)- Do not retry authentication, invalid schema, or deterministic policy failures.
- Prevent both SDK and gateway from independently multiplying attempts.
- Log route, attempt, Retry-After, latency, final result, and duplicate-prevention key.
- 不要重试认证、无效结构定义或确定性策略失败。
- 防止 SDK 与网关各自独立放大重试次数。
- 记录路由、Attempt、Retry-After、延迟、最终结果与防重复密钥。
Verified implementation reference: OpenAI rate limits.
实施参考已根据官方资料核验：OpenAI rate limits。
FAQ
No. Retry only within deadline, budget and provider guidance.
Yes only to an independently eligible and policy-approved route.
Use one retry owner, global concurrency limits, jitter and a circuit breaker.
不应。仅在截止时间、预算与供应商指导内重试。
只能切换到独立合格且策略批准的路由。
使用单一重试负责人、全局并发限制、抖动与熔断。