Two Numbers That Define "Fast"
Users don't experience "latency" as one number — they feel two:
- TTFT (time to first token) — how long before anything appears. This dominates perceived speed in chat and streaming UIs.
- Throughput (tokens per second) — how fast the rest streams out once it starts. This matters most for long outputs.
A model with fast TTFT but modest throughput can feel snappier than a raw-speed champion. Optimize for the one your UX depends on.
What Makes Responses Slow
- Model size & tier. Flagships are slower than mini/flash tiers. A Gemini Flash-Lite or Claude Haiku returns far faster than Opus or GPT-5.5.
- "Thinking"/reasoning modes. Extended reasoning adds latency (and tokens) before the answer. Turn it off for simple tasks.
- Long context. More input to process = slower TTFT. Prune and use RAG.
- Long output. Throughput × length; cap
max_tokensand shape output. - Queueing / rate limits. Hitting limits or provider congestion adds delay — see AI API rate limits.
Speed Varies a Lot by Model
Throughput ranges widely — from ~35 tokens/sec on some configurations to 150+ on fast providers, and the same model can be several times faster on one host than another. Some models ship explicit speed tiers (e.g. MiniMax M2.7's HighSpeed variant ~doubles throughput for ~2x price). If latency is critical, benchmark candidates on your prompts, not published averages.
How to Make Your App Feel Fast
- Stream tokens. Show output as it arrives; this hides throughput behind perceived progress.
- Route small tasks to fast models. Classification, routing, and short replies belong on Flash/Haiku/mini tiers.
- Disable reasoning by default. Enable extended thinking only when a task needs it.
- Trim input, cap output. Both cut TTFT and total time.
- Cache the prefix. Cached context isn't reprocessed, improving TTFT — see prompt caching guide.
- Parallelize independent calls instead of chaining them.
- Add provider fallback so a slow/saturated backend doesn't stall you.
The Speed/Cost/Quality Triangle
Fast, cheap, smart — pick the two your task needs. A cascade gets you most of all three: fast cheap model for the bulk, slower flagship only for the hard 5%. That's the same routing that controls cost — see how to reduce AI API costs and how to choose an LLM.
Benchmark Across Models Easily
Because latency is workload- and provider-specific, test candidates side by side. One gateway (LinkModel) with a shared request shape lets you swap models with a config change and compare TTFT/throughput on your own traffic.
Bottom Line
Optimize for TTFT in interactive UX and throughput for long outputs: stream, route small tasks to fast tiers, disable unnecessary reasoning, trim tokens, and cache. Speed is mostly an architecture choice, not a fixed property of "the API."
Start free with a $1 credit and benchmark latency on your prompts.
