AI operations
A practical guide to LLM API rate limits across OpenAI, Anthropic, Azure OpenAI, Bedrock, and Gemini: TPM, RPM, retry-after, backoff, queues, batching, fallbacks, and throughput planning.
Comparison
| Option | Best for | Strengths | Tradeoffs | Use when |
|---|---|---|---|---|
| Simple backoff | Low-volume apps and occasional 429 responses | Easy to implement and recommended by multiple providers. | Not enough for sustained overload or customer-specific quota planning. | Traffic is modest and failures are rare. |
| Queue and token budget | Production workloads with predictable bursts, batch jobs, or customer tiers | Controls throughput before providers reject traffic. | Adds latency and product decisions about priority and cancellation. | You need fairness, priorities, and predictable capacity. |
| Fallback routing | High-availability products that need alternate models or providers | Keeps some user workflows alive during quota, outage, or latency events. | Requires behavior tests, schema compatibility, and observability. | A 429 or provider incident becomes a customer-facing outage. |
LLM limits can include requests per minute, input tokens per minute, output tokens per minute, requests per day, tokens per day, region quota, deployment quota, or account tier. One dashboard number rarely tells the whole story.
Retries alone do not decide which work matters. Queues let product teams prioritize paying customers, interactive requests, background jobs, and internal batch work differently.
Many rate-limit problems are prompt design problems. Smaller context, caching, batching, summarization, and cheaper routing can reduce pressure before multi-provider architecture is needed.
Decision Rules
Use exponential backoff for occasional 429s.
Use queues and token budgets for sustained production traffic.
Use fallback routing only after schema and quality tests pass.
Plan capacity using successful task throughput, not only raw token limits.
Related Guides
Route traffic when quota, latency, or provider health changes.
OpenEstimate cost and throughput together.
OpenDesign fallback for quota and provider incidents.
OpenReduce repeated token pressure.
OpenMove offline work away from interactive quota.
OpenChinese Archive
Topic Hubs
Industry Pages
FAQ
TPM usually means tokens per minute. Some providers separate input tokens and output tokens, so capacity planning should track both.
Use exponential backoff with jitter for occasional errors, but add queues, token budgets, and fallback routing for sustained production traffic.
No. Limits vary by provider, model, region, tier, deployment, and account. Read current docs and use provider APIs where available instead of hardcoding assumptions.
Source Links
Reference
OpenAI cookbook example for handling rate limits with backoff.
OpenReference
OpenAI help article on preventing rate-limit errors.
OpenReference
Anthropic documentation for Claude API rate limits.
OpenReference
Microsoft documentation for managing Azure OpenAI quota.
OpenReference
AWS documentation for Amazon Bedrock quotas.
OpenThe strongest decision is always local to your workflow. Save the vendor links, define a representative task, record the exact prompt or command, and compare the final evidence instead of the marketing claim.
Return to the AI learning map