LLM API Rate Limits — What to Expect in Production

LLM API rate limits are enforced on two axes simultaneously: requests per minute (RPM) and tokens per minute (TPM). The token ceiling, not the request ceiling, is what breaks most production systems first. A single call with a 200,000-token context window consumes the same TPM budget as roughly 50 calls carrying 4,000-token prompts, which means a service can throw 429 Too Many Requests while sitting at a fraction of its RPM allowance. If you’ve only ever rate-limited traditional web APIs by counting requests, that mental model needs to go before you ship.

How LLM rate limits actually work

Every major provider gates you on multiple dimensions at once. OpenAI enforces RPM, TPM, and (on some endpoints) requests per day and tokens per day; you hit a 429 the moment any one of them is exceeded (OpenAI docs). Anthropic publishes a per-model table with separate input-token and output-token minute buckets (Claude docs). Google’s Gemini limits span RPM, TPM, and RPD per model tier, and Google explicitly notes those limits “are not guaranteed” and can shift (Gemini docs). xAI’s Grok API gates on requests per second and tokens per minute instead of RPM — your per-second cap is derived from your per-minute budget so you can’t dump a full minute’s quota into a single burst — and tiers unlock automatically on cumulative spend and never downgrade once reached (xAI docs).

The practical takeaway: your bottleneck is almost always tokens, not requests. Teams running long-context RAG pipelines or agent loops routinely burn through TPM at low request volumes because each call drags a bloated context window along with it. Thirty requests a minute with 150K-token contexts pushes 4.5M tokens per minute — enough to trip most tier-2 accounts instantly.

Which provider’s limits caught me off guard

The one that bit me hardest was Google Gemini’s free tier, and the failure was less about the number and more about the unpredictability of the number. On December 7, 2025, Google slashed free-tier quotas without an announcement, an email, or a changelog entry. Gemini Flash’s requests-per-day dropped from 250 to 20, a 92% cut. The first signal anyone got was a wall of 429s in their logs.

That incident crystallized a rule I now treat as non-negotiable: free tiers are for prototyping, never for anything a user depends on. Google literally documents that its limits aren’t guaranteed, and it acted on that clause with zero warning. A free tier is a testing sandbox whose floor can drop out between one deploy and the next. If a workload matters, it lives on a paid tier with a contractual quota you can actually plan against — and even then, you build for the limit to change.

The second flavor of surprise is the noisy-neighbor problem inside shared org accounts. Rate limits are usually scoped to the project or organization, not the individual app. A developer running a batch backfill, a CI job chewing through a queue, or an agent stuck in a retry loop can throttle the entire org — and your well-behaved production chatbot starts returning 429s despite doing nothing wrong. If you share a quota across teams, you need workload-level budgets, or you’re one runaway script away from an outage you didn’t cause.

Should you build retry logic on day one?

My rule of thumb is simple: retry logic ships on day one, always. Provider selection and budget caps can wait until you have real traffic data.

Retry logic is cheap to add up front and expensive to bolt on after an incident. Exponential backoff that respects the Retry-After header is maybe 20 lines of code, and there’s no scenario where you regret having it. OpenAI’s own cookbook recommends exponential backoff with jitter as the baseline pattern for handling 429s gracefully (OpenAI Cookbook). You do not want to be writing that code at 2 a.m. while your queue backs up.

What I don’t over-invest in early is token-budget accounting and multi-provider fallback. Those are real problems, but they’re tuning problems — you need production token distributions to size them correctly, and premature abstraction usually produces the wrong caps. The community pattern matches this almost exactly: most teams add request rate limiting first, add token rate limiting after their first surprise invoice, and add hard budget caps after the second. Skip straight to token-aware limiting if you’re doing long-context work, but everyone should have backoff from commit one.

Concretely, a day-one client should:

1. Catch 429 (and 529 on Anthropic) and back off exponentially — start at ~1s, cap at ~60s, add jitter.
2. Honor the Retry-After header when present instead of guessing.
3. Queue and smooth bursts rather than firing requests machine-gun style — bursty traffic is what actually trips limits, not sustained load.
4. Cap total retries so a stuck request doesn’t become a noisy-neighbor incident of its own.

When rate limits force a provider switch

The tipping point for switching providers is rarely the raw limit number — it’s the friction and speed of getting the limit raised. If a provider makes you file a ticket, wait days, and justify your usage before a quota bump, while a competitor auto-scales your tier as spend increases, that operational gap will eventually cost you a launch window.

The other tipping point is opacity. As of 2026, OpenAI and Google largely push you to a dashboard to see your actual limits, and Google won’t guarantee them. Anthropic and xAI are the exceptions — both still publish a full per-model rate-limit table you can plan against. When you can’t answer “what is my ceiling and when will I hit it?” without logging into a console, capacity planning becomes guesswork — and guesswork is a legitimate reason to migrate to a provider whose limits are documented and predictable. If you’re weighing providers on this axis, our breakdown of the best LLM APIs for developers compares how they handle quota transparency and tier progression.

The migration itself is easier than the fear suggests because most providers now expose OpenAI-compatible endpoints. The hard part is re-validating prompt behavior and token accounting on the new model, not rewiring the HTTP client.

Architecting for limits before you hit them

If you operate at scale, don’t leave enforcement to the provider. Put a gateway in front of your LLM calls and enforce token budgets yourself. Azure API Management, for example, ships an llm-token-limit policy that meters TPM per subscription key and returns 429 with a Retry-After header before the upstream provider ever sees the traffic (Microsoft Learn). That’s how you solve the noisy-neighbor problem: give each workload its own budget so a runaway batch job can’t starve production.

The habits that separate teams who sleep well from teams who get paged:

• Limit by tokens, not requests. Track TPM per workload and alert at 80% of your provider ceiling.
• Never put production on a free tier. The Gemini incident proved free quotas can be cut 90%+ overnight with no notice.
• Isolate workloads. Batch jobs, CI, agents, and user-facing traffic should each have their own budget so one can’t take down the others.

Rate limits aren’t an edge case you handle after launch. They’re a load-bearing part of any serious LLM integration. Treat 429 as an expected response code, not an exception, and design accordingly.