OpenRouter
Model routing for compatibility and fallback checks.
INCIDENT RUNBOOK
OpenAI Rate Limit Fallback
Handle OpenAI rate limits with bounded retries, explicit fallback rules, and visible provider changes.
Published: March 16, 2026 · By the QVeris Team
On this page
Summary
Read the native signal
Preserve status, request ID and retry or limit headers when available.
Queue or back off first
If the deadline allows, waiting can be safer than changing execution semantics.
Use eligible alternates
Check model, tools, region, policy, data and budget before switching.
Prevent amplification
Coordinate SDK, gateway and workflow retries under one request budget.
429 response options
A rate limit may apply to a project, organization, user, model, deployment, request rate or token budget. Interpret that scope before deciding whether another route is actually independent. A second model under the same project quota may fail for the same reason, while a different deployment can still be unsuitable if it changes the context window, tool schema or data-residency contract.
Choose among fail-fast, a bounded queue, jittered backoff or a verified alternate based on the parent deadline and workload contract. Interactive requests usually favor a quick, explainable response; asynchronous jobs can wait only while queue age remains inside their service-level objective. Record the 429 scope, provider hint, selected branch and remaining budget so the decision can be explained later.
Rate-limit response paths
| Path | Best fit | Verify before choosing |
|---|---|---|
| Fail fast | Interactive workload cannot wait or substitute. | Return a controlled error with retry guidance and request ID. |
| Queue | Deadline and queue capacity are known. | Bound queue age, depth, cancellation and tenant fairness. |
| Backoff | Same endpoint is expected to recover within budget. | Honor hints, add jitter and stop at the parent deadline. |
| Alternate | A separate eligible route preserves the contract. | Do not hop quotas, bypass policy or duplicate tool actions. |
Safeguards for every branch
Global budget
Count all attempts, waiting, tokens and spend under one parent request.
Idempotency
Deduplicate tool calls and downstream actions before retry.
Quota policy
Enforce tenant and project budgets across every key and provider.
Audit trail
Record the 429, decision, alternate, attempt and recovery outcome.
Exercise the 429 runbook
A fallback design is not proven by a successful happy-path request. Test it under realistic concurrency and with the same SDK, gateway and worker retry settings used in production. The important result is not merely “a response arrived,” but that recovery stayed within the end-to-end deadline without multiplying attempts or changing the answer contract.
- Inject 429s with and without retry hints at project, model and token-budget scopes.
- Verify queue depth and age limits, jitter, cancellation propagation and request deadlines.
- Confirm alternate routes preserve required tools, structured output, data region and safety policy.
- Test simultaneous failures so a shared fallback does not become the next bottleneck.
- Review alerts, trace evidence, duplicate suppression and controlled recovery after the incident closes.
Centralize rate-limit decisions
The gateway should normalize the 429, retain the provider's native error details, check the parent budget and select fail-fast, queue, backoff or an eligible alternate. Applications then submit one request contract instead of embedding different retry loops in every SDK integration. Central policy also makes tenant fairness, concurrency ceilings and circuit-breaker state consistent across services.
Assign exactly one retry owner. If a provider SDK, gateway and job worker each retry twice, one user request can fan out into many billable attempts and worsen the limit. Pass a shared attempt count, absolute deadline and trace ID through every layer; when the budget is exhausted, return a controlled result rather than starting another hidden loop.
Production rule: never treat extra keys or providers as a way to evade an enforced quota. Alternate routes are for contract-preserving resilience, not quota hopping.
Keep external actions safe during retries
QVeris can call external APIs, tools, services and live data during an agent workflow. A model request may be safe to repeat, while the tool action it already triggered—placing an order, sending a message or updating a record—may not be. Keep the inference attempt and the external action as separate states rather than replaying the entire workflow after every 429.
Propagate the parent trace, action ID and idempotency key into the tool layer, then persist whether the action was planned, submitted, confirmed or rejected. On recovery, resume from the last verified state and re-read time-sensitive data when freshness matters. This prevents a rate-limit fallback from silently duplicating a real-world side effect or acting on stale evidence.
Fallback only after classifying the 429
A 429 can mean a short burst, account quota, per-model capacity, or policy. Respect Retry-After when present, consume a shared attempt budget, and fall back only to a model that has passed the same capability and quality gates.
Bounded fallback loop
async def call_with_fallback(request, routes, max_attempts=2):
errors = []
for attempt, route in enumerate(routes[:max_attempts], start=1):
try:
return await route.client.call(
model=route.model,
request=request,
timeout=route.timeout,
idempotency_key=request.id,
)
except RateLimitError as error:
errors.append((route.name, "429", error.retry_after))
if error.retry_after and error.retry_after <= 2:
await asyncio.sleep(error.retry_after)
continue
raise AllRoutesLimited(errors)- Do not retry authentication, invalid schema, or deterministic policy failures.
- Prevent both SDK and gateway from independently multiplying attempts.
- Log route, attempt, Retry-After, latency, final result, and duplicate-prevention key.
Verified implementation reference: OpenAI rate limits.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
Should a 429 always retry?
No. Retry only within deadline, budget and provider guidance.
Can I switch providers?
Yes only to an independently eligible and policy-approved route.
How do I stop retry storms?
Use one retry owner, global concurrency limits, jitter and a circuit breaker.