Avoid OpenAI Rate Limits
Build Resilience, Do Not Evade Quotas应对 OpenAI 限流：建设系统韧性，不规避配额
Rate limits protect service stability. Reliable applications shape demand, honor provider signals and request more capacity through official channels; they do not rotate keys to evade quotas.
Rate Limit 用于保护服务稳定。可靠应用会塑造需求、遵循供应商信号，并通过官方渠道申请更高容量；不会轮换密钥来规避配额。
TL;DR
Use admission control, priority queues and per-workload request and token budgets.
Read current rate-limit and retry guidance; back off with jitter.
Set a deadline, maximum attempts and one retry owner across all layers.
Monitor legitimate growth and request higher limits through approved processes.
使用准入控制、优先队列与按负载划分的请求和 Token 预算。
读取当前限流与重试指导，并使用带 Jitter 的退避。
设置截止时间、最大尝试次数，并让所有层只有一个重试 Owner。
监控合理增长，并通过获批流程申请更高限制。
What a rate-limit response means限流响应意味着什么
A rate-limit error can reflect request rate, token throughput, concurrent work, project or organization quota, model capacity or spend constraints. Classify the returned error at the scope the provider reports; changing models may not help when both routes share the same project budget. Treat the native error, retry guidance and current provider documentation as the source of truth for that request.
限流错误可能反映请求速率、Token 吞吐、并发工作、Project 或 Organization 配额、模型容量或支出约束。应按照供应商报告的作用范围分类；如果两条路由共享同一 Project 预算，换模型也未必有效。该请求应以原生错误、重试提示与当前供应商文档为准。
The most reliable way to avoid rate-limit incidents is to shape demand before the API call. Estimate request and token demand, reserve capacity for latency-sensitive workloads, cap tenant concurrency and queue only work whose deadline can tolerate waiting. Batch or schedule deferrable jobs rather than letting them compete with interactive traffic during a burst.
避免限流事故最可靠的方法，是在 API Call 前塑造需求。估算请求与 Token 需求，为延迟敏感型负载保留容量，限制租户并发，只让截止时间允许等待的任务进入队列。可延后任务应批处理或调度执行，避免在突发期间与交互流量竞争。
A retry consumes capacity too. Immediate synchronized retries amplify an incident and can keep a recovering service overloaded. Respect provider timing hints, add randomized exponential backoff, use one retry owner and stop when the parent deadline, attempt limit or spend budget is exhausted. If work must be shed, return a controlled response with a request ID and realistic retry guidance.
重试同样消耗容量。立即同步重试会放大事故，并让正在恢复的服务持续过载。应遵循供应商时间提示，使用随机指数退避，指定唯一重试负责人，并在父截止时间、尝试次数或支出预算耗尽时停止。必须降载时，返回带 Request ID 和现实重试建议的受控响应。
Compliant response options合规响应选项
| Option选项 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Queue and backoff排队与退避 | Interactive work can wait inside a bounded deadline.交互工作可在有界截止时间内等待。 | Queue age, cancellation, priority, jitter and retry ownership.核对队列年龄、取消、优先级、Jitter 与重试归属。 |
| Batch eligible work批处理合格负载 | Offline jobs can use an officially supported batch path.离线任务可使用官方支持的批处理路径。 | Feature support, completion window, result handling and current terms.核对功能支持、完成窗口、结果处理与当前条款。 |
| Cache safe results缓存安全结果 | Identical or reusable deterministic work has an approved cache policy.相同或可复用的确定性工作有获批缓存策略。 | Freshness, privacy, tenant isolation, invalidation and correctness.核对新鲜度、隐私、租户隔离、失效与正确性。 |
| Eligible fallback合格回退 | An alternate model or provider independently satisfies the workload.备选模型或供应商独立满足负载。 | User intent, policy, capabilities, data terms and semantic differences.核对用户意图、策略、能力、数据条款与语义差异。 |
Reliability controls可靠性控制
Reject or defer excess work before it consumes scarce upstream capacity.
Coordinate request, token, concurrency and spend limits across replicas.
Drop optional work and return controlled errors under sustained pressure.
Track saturation, queue age, rate-limit errors, retries and exhausted budgets.
在过量工作消耗稀缺上游容量前拒绝或延后。
在副本间协调请求、Token、并发与支出限制。
持续压力下丢弃可选工作并返回受控错误。
追踪饱和度、队列年龄、限流错误、重试与预算耗尽。
Test rate-limit behavior before an incident事故前测试限流行为
Use a safe simulator or approved test environment; do not deliberately exhaust a production quota to see what happens. Reproduce both short bursts and sustained demand with the prompt sizes, output lengths and tenant distribution the application actually produces.
应使用安全 Simulator 或经过批准的测试环境，不能为了观察结果而故意耗尽生产配额。既要重现短时突发，也要重现持续需求，并使用应用真实的 Prompt 大小、输出长度与租户分布。
- Inject rate-limit responses with and without retry hints at model, project and token-budget scopes.
- Verify header-aware bounded backoff, one retry owner and a shared attempt budget.
- Measure queue depth and age, cancellation propagation, shedding and tenant fairness.
- Confirm priority traffic retains reserved capacity while background jobs degrade first.
- Exercise official capacity escalation, controlled degradation and recovery without a retry surge.
- 在模型、Project 与 Token 预算等范围注入带与不带重试提示的限流响应。
- 验证 Header 感知的有界退避、唯一重试负责人和共享尝试预算。
- 测量队列深度与年龄、取消传播、降载和租户公平性。
- 确认优先流量仍拥有保留容量，后台任务先降级。
- 演练官方容量申请、受控降级与不会引发重试突增的恢复过程。
Centralize demand and retry decisions集中需求与重试决策
Traffic first enters admission control, where the system estimates demand and checks tenant, workload and global budgets. Accepted asynchronous work enters a bounded priority queue with maximum age and cancellation support; interactive work proceeds only if capacity and deadline remain. A distributed budget service grants request, token and concurrency capacity atomically enough to prevent every worker from believing the same quota is free.
流量先进入 Admission Control，系统在此估算需求并检查租户、工作负载与全局预算。被接受的异步任务进入带最大等待年龄与取消支持的有界优先队列；交互任务只有在容量与截止时间仍允许时才继续。分布式 Budget Service 以足够原子的方式分配请求、Token 与并发容量，避免每个 Worker 都以为同一份配额尚未占用。
One execution layer calls the API, preserves native error evidence and owns retries. Success returns normally; a 429 triggers a policy choice among bounded wait, shedding or an independently eligible alternate. Telemetry feeds observed consumption back to admission control, while reconciliation compares estimates with provider usage so local budgets do not drift indefinitely.
单一执行层负责调用 API、保留原生错误证据并拥有重试权。成功正常返回；429 则根据策略选择有界等待、降载或独立合格备用项。Telemetry 将观测用量反馈给 Admission Control，对账再把本地估算与供应商用量比较，避免本地预算长期漂移。
Production rule: never rotate keys, shard accounts or change identity to evade an enforced quota.
生产规则：绝不能通过轮换 Key、拆分 Account 或更改身份规避已执行配额。
Protect external actions during backoff and retry在退避与重试期间保护外部动作
QVeris may call external APIs, tools, services and live data before, during or after model decisions. A rate-limited inference call and a completed external action are different states. Persist whether an action was planned, submitted, confirmed or rejected so a queued or fallback model request does not replay the whole workflow.
QVeris 可能在模型决策之前、期间或之后调用外部 API、工具、服务与实时数据。被限流的推理调用与已经完成的外部动作属于不同状态。应持久化动作处于已规划、已提交、已确认还是已拒绝，避免排队或故障切换后的模型请求重放整个工作流。
Propagate the parent deadline and cancellation signal, but use separate action IDs and idempotency keys for tool execution. On recovery, resume from the last verified state and refresh live data only when its freshness window has expired. This prevents both duplicate side effects and decisions made from evidence that became stale while waiting.
父截止时间与 Cancellation Signal 可以继续传播，但工具执行应使用独立 Action ID 与幂等密钥。恢复时从最后一个已验证状态继续，只有实时数据的新鲜度窗口已过期时才刷新，既避免重复副作用，也避免基于等待期间已经过期的证据做决策。
Calculate admission before a request reaches the model在请求到达模型前计算准入
Avoiding rate limits begins with workload shaping, not retries. Estimate input plus reserved output tokens, enforce per-tenant concurrency, and queue only work whose deadline can still be met. Retry logic is the final guard, not the capacity plan.
避免限流首先依赖工作负载整形，而不是重试。估算输入加预留输出 Token，执行每租户并发控制，并且只排队仍能满足截止时间的任务。重试是最后防线，而不是容量计划。
def admit(request, tenant, limiter):
estimated = estimate_input_tokens(request) + request.max_output_tokens
if request.deadline_ms <= now_ms():
return Reject("deadline_expired")
if not limiter.concurrent.try_acquire(tenant):
return QueueOrReject("tenant_concurrency")
if not limiter.tokens.try_reserve(tenant, estimated):
limiter.concurrent.release(tenant)
return QueueOrReject("token_budget")
return Permit(reserved_tokens=estimated)- Separate RPM, TPM, concurrency, spend, and provider-capacity signals.
- Use bounded queues with deadlines so traffic spikes do not become latency incidents.
- Reconcile reserved tokens with actual usage and release capacity on cancellation.
- 分别处理 RPM、TPM、并发、支出与供应商容量信号。
- 使用带截止时间的有界队列，避免流量峰值变成延迟事故。
- 用实际用量对账预留 Token，并在取消时释放容量。
Verified implementation reference: OpenAI rate limits.
实施参考已根据官方资料核验：OpenAI rate limits。
FAQ
No. Do not evade quotas; follow provider terms and request capacity officially.
No. Retry only when guidance, deadline, budget and idempotency permit it.
Only when the alternate is independently eligible and policy-approved.
不可以。不要规避配额；遵循供应商条款并通过官方方式申请容量。
不应。只有指导、截止时间、预算与幂等允许时才重试。
只有备选独立合格且策略批准时才允许。