Avoid OpenAI Rate Limits
Build Resilience, Do Not Evade Quotas
应对 OpenAI 限流：建设系统韧性，不规避配额
Rate limits protect service stability. Reliable applications shape demand, honor provider signals and request more capacity through official channels; they do not rotate keys to evade quotas.
Rate Limit 用于保护服务稳定。可靠应用会塑造需求、遵循供应商信号，并通过官方渠道申请更高容量；不会轮换密钥来规避配额。
TL;DR
Use admission control, priority queues and per-workload request and token budgets.
Read current rate-limit and retry guidance; back off with jitter.
Set a deadline, maximum attempts and one retry owner across all layers.
Monitor legitimate growth and request higher limits through approved processes.
使用准入控制、优先队列与按负载划分的请求和 Token 预算。
读取当前限流与重试指导，并使用带 Jitter 的退避。
设置截止时间、最大尝试次数，并让所有层只有一个重试 Owner。
监控合理增长，并通过获批流程申请更高限制。
What a rate-limit response means 限流响应意味着什么
A rate-limit error can reflect request, token, concurrent, project, organization, model or spend constraints. Treat the returned error and current provider documentation as the source of truth for that request.
限流错误可能反映请求、Token、并发、Project、Organization、模型或 Spend 约束。该请求应以返回错误与当前供应商文档为准。
A retry consumes capacity too. Immediate synchronized retries can amplify an incident. Queue or shed work, respect retry timing, add randomized backoff and stop when the parent deadline or budget is exhausted.
重试同样消耗容量。立即同步重试会放大事故。应排队或降载、尊重重试时间、加入随机退避，并在父截止时间或预算耗尽时停止。
Compliant response options 合规响应选项
| Option 选项 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Queue and backoff 排队与退避 | Interactive work can wait inside a bounded deadline. 交互工作可在有界截止时间内等待。 | Queue age, cancellation, priority, jitter and retry ownership. 核对队列年龄、取消、优先级、Jitter 与重试归属。 |
| Batch eligible work 批处理合格负载 | Offline jobs can use an officially supported batch path. 离线任务可使用官方支持的批处理路径。 | Feature support, completion window, result handling and current terms. 核对功能支持、完成窗口、结果处理与当前条款。 |
| Cache safe results 缓存安全结果 | Identical or reusable deterministic work has an approved cache policy. 相同或可复用的确定性工作有获批缓存策略。 | Freshness, privacy, tenant isolation, invalidation and correctness. 核对新鲜度、隐私、租户隔离、失效与正确性。 |
| Eligible fallback 合格回退 | An alternate model or provider independently satisfies the workload. 备选模型或供应商独立满足负载。 | User intent, policy, capabilities, data terms and semantic differences. 核对用户意图、策略、能力、数据条款与语义差异。 |
Reliability controls 可靠性控制
Reject or defer excess work before it consumes scarce upstream capacity.
Coordinate request, token, concurrency and spend limits across replicas.
Drop optional work and return controlled errors under sustained pressure.
Track saturation, queue age, rate-limit errors, retries and exhausted budgets.
在过量工作消耗稀缺上游容量前拒绝或延后。
在副本间协调请求、Token、并发与支出限制。
持续压力下丢弃可选工作并返回受控错误。
追踪饱和度、队列年龄、限流错误、重试与预算耗尽。
Test rate-limit behavior before an incident 事故前测试限流行为
- Replay bursts and sustained load against a safe test environment or simulator.
- Inject rate-limit responses and verify header-aware bounded backoff.
- Confirm queues, cancellation, shedding and priority remain fair by tenant.
- Exercise official capacity escalation and a controlled degradation runbook.
- 在安全测试环境或 Simulator 中回放突发与持续负载。
- 注入限流响应，验证 Header 感知的有界退避。
- 确认队列、取消、降载与优先级按租户保持公平。
- 演练官方容量申请与受控降级运行手册。
Centralize demand and retry decisions 集中需求与重试决策
Traffic enters admission control, then a priority queue. A distributed budget service grants request and token capacity. One execution layer calls the API and interprets native responses. Success returns normally; rate limits trigger bounded backoff, shedding or a policy-approved alternate.
流量先进入准入控制，再进入优先队列；分布式 Budget Service 分配请求与 Token 容量；单一执行层调用 API 并解释原生响应；成功正常返回，限流则触发有界退避、降载或策略批准的备选。
Production rule: never rotate keys, shard accounts or change identity to evade an enforced quota.
生产规则：绝不能通过轮换密钥、拆分 Account 或更改身份规避已执行配额。
Protect external actions during backoff and retry 在退避与重试期间保护外部动作
QVeris may call external APIs, tools, services and live data after model decisions. Propagate cancellation and idempotency keys so queued, retried or fallback inference cannot duplicate real-world actions.
QVeris 可能在模型决策后调用外部 API、工具、服务与实时数据。应传播 Cancellation 与 Idempotency 密钥，避免排队、重试或回退推理重复真实世界动作。
Calculate admission before a request reaches the model 在请求到达模型前计算准入
Avoiding rate limits begins with workload shaping, not retries. Estimate input plus reserved output tokens, enforce per-tenant concurrency, and queue only work whose deadline can still be met. Retry logic is the final guard, not the capacity plan.
避免限流首先依赖工作负载整形，而不是重试。估算输入加预留输出 Token，执行每租户并发控制，并且只排队仍能满足截止时间的任务。重试是最后防线，而不是容量计划。
def admit(request, tenant, limiter):
estimated = estimate_input_tokens(request) + request.max_output_tokens
if request.deadline_ms <= now_ms():
return Reject("deadline_expired")
if not limiter.concurrent.try_acquire(tenant):
return QueueOrReject("tenant_concurrency")
if not limiter.tokens.try_reserve(tenant, estimated):
limiter.concurrent.release(tenant)
return QueueOrReject("token_budget")
return Permit(reserved_tokens=estimated)
- Separate RPM, TPM, concurrency, spend, and provider-capacity signals.
- Use bounded queues with deadlines so traffic spikes do not become latency incidents.
- Reconcile reserved tokens with actual usage and release capacity on cancellation.
- 分别处理 RPM、TPM、并发、支出与供应商容量信号。
- 使用带截止时间的有界队列，避免流量峰值变成延迟事故。
- 用实际用量对账预留 Token，并在取消时释放容量。
Verified implementation reference: OpenAI rate limits.
实施参考已根据官方资料核验：OpenAI rate limits。
The Production Operating Model for OpenAI rate-limit protectionOpenAI 限流保护的生产运营模型
For Avoid OpenAI Rate Limits, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“应对 OpenAI Rate Limit”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory request and token quotas, organization and project scope, burst behavior, concurrency, admission queues, backoff, retry budgets, priority classes, and independently eligible fallback. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点请求与 Token 配额、组织与项目 Scope、突发行为、并发、准入队列、退避、重试预算、优先级类别和独立合格的故障切换。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For Avoid OpenAI Rate Limits, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“应对 OpenAI Rate Limit”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate Avoid OpenAI Rate Limits, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“应对 OpenAI Rate Limit”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating Avoid OpenAI Rate Limits, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“应对 OpenAI Rate Limit”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
No. Do not evade quotas; follow provider terms and request capacity officially.
No. Retry only when guidance, deadline, budget and idempotency permit it.
Only when the alternate is independently eligible and policy-approved.
不可以。不要规避配额；遵循供应商条款并通过官方方式申请容量。
不应。只有指导、截止时间、预算与幂等允许时才重试。
只有备选独立合格且策略批准时才允许。