AI API Latency Guide: TTFT, Throughput & How to Make It Fast

Two Numbers That Define "Fast"

Users don't experience "latency" as one number — they feel two:

TTFT (time to first token) — how long before anything appears. This dominates perceived speed in chat and streaming UIs.
Throughput (tokens per second) — how fast the rest streams out once it starts. This matters most for long outputs.

A model with fast TTFT but modest throughput can feel snappier than a raw-speed champion. Optimize for the one your UX depends on.

What Makes Responses Slow

Model size & tier. Flagships are slower than mini/flash tiers. A Gemini Flash-Lite or Claude Haiku returns far faster than Opus or GPT-5.5.
"Thinking"/reasoning modes. Extended reasoning adds latency (and tokens) before the answer. Turn it off for simple tasks.
Long context. More input to process = slower TTFT. Prune and use RAG.
Long output. Throughput × length; cap max_tokens and shape output.
Queueing / rate limits. Hitting limits or provider congestion adds delay — see AI API rate limits.

Speed Varies a Lot by Model

Throughput ranges widely — from ~35 tokens/sec on some configurations to 150+ on fast providers, and the same model can be several times faster on one host than another. Some models ship explicit speed tiers (e.g. MiniMax M2.7's HighSpeed variant ~doubles throughput for ~2x price). If latency is critical, benchmark candidates on your prompts, not published averages.

How to Make Your App Feel Fast

Stream tokens. Show output as it arrives; this hides throughput behind perceived progress.
Route small tasks to fast models. Classification, routing, and short replies belong on Flash/Haiku/mini tiers.
Disable reasoning by default. Enable extended thinking only when a task needs it.
Trim input, cap output. Both cut TTFT and total time.
Cache the prefix. Cached context isn't reprocessed, improving TTFT — see prompt caching guide.
Parallelize independent calls instead of chaining them.
Add provider fallback so a slow/saturated backend doesn't stall you.

The Speed/Cost/Quality Triangle

Fast, cheap, smart — pick the two your task needs. A cascade gets you most of all three: fast cheap model for the bulk, slower flagship only for the hard 5%. That's the same routing that controls cost — see how to reduce AI API costs and how to choose an LLM.

Benchmark Across Models Easily

Because latency is workload- and provider-specific, test candidates side by side. One gateway (LinkModel) with a shared request shape lets you swap models with a config change and compare TTFT/throughput on your own traffic.

Bottom Line

Optimize for TTFT in interactive UX and throughput for long outputs: stream, route small tasks to fast tiers, disable unnecessary reasoning, trim tokens, and cache. Speed is mostly an architecture choice, not a fixed property of "the API."

Start free with a $1 credit and benchmark latency on your prompts.