LLM API Latency Benchmarks — Which Is Actually Fastest?

Every LLM provider claims to be fast. Their benchmark pages show impressive tokens-per-second charts and sub-second latency numbers that fall apart the moment you route production traffic through them. The gap between marketing latency and the latency your users actually feel is where most teams get burned.

This article ranks the major LLM APIs by real latency behavior — not vendor-supplied figures — and tells you which one to reach for when speed is a hard requirement.

What “LLM API latency” actually measures

Latency for an LLM API is not one number. It breaks into two components that behave very differently:

• Time to first token (TTFT): how long you wait before any output arrives. This dominates perceived responsiveness in chat and streaming UIs.
• Tokens per second (TPS): how fast the rest of the response streams once it starts. This dominates total completion time on long outputs.

A model can win one and lose the other. GPT-5.5 in standard mode streams at roughly 92 tokens per second with a TTFT around 1.1 seconds — respectable until you flip on extended thinking, which pushes TTFT past 27 seconds. Claude Haiku 4.5, by contrast, returns its first token in about 500ms. That’s the difference between an app that feels alive and one that feels broken, even before either model finishes its answer.

The second trap is averaging. If you only track median (P50) latency, you’re ignoring the users having a genuinely bad time. A P50 of 500ms with a P99 of 4 seconds means 1 in 100 requests takes eight times longer than typical. On a high-traffic endpoint, that’s thousands of frustrated users a day, invisible on your dashboard.

Which LLM API is fastest?

On raw TTFT, Groq is still the clear winner among mainstream inference APIs — roughly 120–180ms median time to first token and around 315 TPS on Llama-class models, which is 3–5x faster than OpenAI or Anthropic. Cerebras has pulled ahead on raw throughput (2,100 TPS on Llama 3.3 70B, and it recently ran GPT-5.6 Sol at 750 TPS), though its API access is narrower. SambaNova (~435 TPS on MiniMax M2.7) rounds out the custom-hardware tier.

The rankings for practical use look like this:

Provider	Median TTFT	TPS	Notes
Groq	~120–180ms	~315	Fastest mainstream TTFT, limited model catalog
Cerebras	low	~2,100	Highest raw throughput, narrow API access
SambaNova	~150ms	~435	Custom hardware, close second on TTFT
Claude Haiku 4.5	~500ms	good	Best consistency (1.8x P95/P50 ratio)
Claude Sonnet 4.6	~380–800ms	~40–95	Best speed-to-capability ratio in Claude’s lineup
Gemini 3.5 Flash	~700ms	~184–251	Strong cost-latency balance
GPT-5.5 (standard)	~1,100ms	~92	Jumps to 27s+ TTFT with reasoning on
Claude Opus 5 (medium)	~6.7s	~54	Frontier reasoning; TTFT climbs with effort level

Groq’s advantage on time-to-first-token is real and reproducible. If your only metric is “how fast does the first token show up,” nothing mainstream beats it right now. For a closer look at what building on it actually involves, see our Groq API developer review.

The consistency problem nobody benchmarks

Speed means nothing if it’s unpredictable. This is where the rankings shift.

Anthropic is the most consistent provider, with a P95/P50 ratio of just 1.8x. Slow Claude requests land at only 1.8 times the median — surprise stalls are rare. For user-facing work, low variance often beats a lower average, because users notice the outliers, not the typical case.

The cautionary example is DeepSeek: a strong ~300ms median TTFT that collapses to a ~2,500ms P95. On paper it looks competitive with the fast tier. In production it’s a tail-latency disaster — most requests feel snappy, and then one in twenty hangs for two and a half seconds. If you’d shipped based on the median alone, you’d have no idea why your complaint volume was climbing.

The lesson: always benchmark P95 and P99 with your own prompts, at your own concurrency, before committing. Vendor pages report best-case medians under ideal load. Your traffic pattern is not their traffic pattern.

Where Groq’s speed advantage disappears

Groq being fastest doesn’t mean Groq is always the right answer, and the reasons are worth being blunt about.

The catalog problem. Groq doesn’t host GPT-5.6, Claude, or Gemini. If your product depends on a specific frontier model — because of its reasoning quality, tool-use reliability, or vision support — Groq’s 120ms TTFT is irrelevant. You either stay on the original provider or front everything with a router like OpenRouter and accept the extra hop. Cerebras is starting to crack this (it ran GPT-5.6 Sol at 750 TPS), but that kind of cross-provider hosting is still the exception.

The bottleneck-elsewhere problem. Shaving 2 seconds off token generation does nothing if the real delay lives somewhere else in your stack. A retrieval-augmented pipeline that spends 800ms on vector search, 400ms on reranking, and 600ms on a database round-trip before the model even sees the prompt will feel slow no matter how fast inference is. Reasoning models make this worse — they think longer and emit more tokens, so a “fast” provider generating 1,500 reasoning tokens can still lose to a slower provider generating 300. Inference speed is only worth chasing once you’ve confirmed inference is your bottleneck.

The output-volume problem. Latency and cost interact in ways benchmarks hide. If Gemini 3.5 Flash produces 1.8x more output tokens than GPT-5.5 for the same task, it can finish “faster” per token while costing more and taking longer end-to-end. Measure total wall-clock time to a complete, usable answer — not TPS in isolation.

What to actually reach for in production

Here’s a direct answer, because “it depends” isn’t one:

For real-time, interactive workloads — chat, autocomplete, voice, anything where the user is watching the cursor — reach for Groq, provided one of its hosted open models is good enough for the task. You give up the frontier-model catalog in exchange for latency you can’t get anywhere else. For most classification, extraction, summarization, and lightweight-agent tasks, that tradeoff is easy.

For user-facing work that needs a frontier model, reach for Claude Haiku 4.5, Claude Sonnet 4.6, or Gemini 3.5 Flash. Haiku and Gemini Flash both hold TTFT under a second on medium prompts. Sonnet 4.6 sits in a sweet spot — roughly 380–800ms TTFT with meaningfully better reasoning than Haiku, and it’s the fastest Claude model that still handles complex tool use well. Claude’s 1.8x P95/P50 ratio makes it the safer pick across the lineup when tail latency will generate support tickets.

For deep analysis where capability outranks speed — complex reasoning, long-context synthesis, high-stakes tool use — use Claude Opus 5, Claude Sonnet 5, or GPT-5.6 Sol and stop optimizing for latency. These models use adaptive reasoning that trades TTFT for output quality — Opus 5 at medium effort posts a ~6.7s TTFT, and Sonnet 5 at high effort can hit ~8s. Users tolerate a slower response when the output is visibly doing more work. The key lever here is reasoning effort: Claude’s adaptive thinking lets you dial effort from low (~2s TTFT) to max depending on task complexity, so you’re not paying the full latency penalty on every request.

Sophisticated teams don’t pick one. They run a split architecture: Groq or another fast provider for real-time interactions, and Claude or GPT for the heavy lifting behind the scenes. A router in front lets you switch per-request based on what the task actually needs.

The one benchmark that matters

Ignore the vendor charts. Build a script that fires your real prompts at each candidate provider, at your expected concurrency, and records P50, P95, and P99 TTFT plus total completion time. Run it for at least a few hundred requests across different times of day — provider latency drifts with load.

The fastest LLM API is Groq. The right LLM API is the one that clears your latency bar on your prompts, at your tail percentiles, hosting a model good enough for the job. Those are frequently not the same provider — and the only way to know is to measure it yourself.