AI API Rate Limits Explained: RPM, TPM, 429s & How to Handle Them

Why Rate Limits Exist (and Bite)

Every AI API caps how much you can send in a window to protect shared capacity. You'll usually see two limits:

• RPM — requests per minute: how many calls you can make.
• TPM — tokens per minute: how many tokens (in + out) you can process.

Hit either and the API returns HTTP 429 (Too Many Requests). Limits scale with your account tier/spend, and they differ per model — the newest flagship often has the tightest limits at launch. Some providers also cap concurrent requests and image/video jobs separately from text.

Why You're Getting 429s

Common causes, roughly in order:

1. Bursty traffic — a spike blows past your per-minute window even if your average is fine.
2. Big prompts — long context eats your TPM fast; a few 200K-token calls can exhaust it.
3. Low account tier — new/low-spend accounts start conservative.
4. Parallel workers — concurrency multiplies your request rate without you noticing.
5. Retry storms — naive retries on 429 make the problem worse.

How to Handle Them Properly

1. Exponential backoff with jitter. On a 429, wait, then retry with increasing, randomized delays — never hammer immediately.

import time, random, requests
 
def call_with_backoff(fn, max_retries=6):
    for attempt in range(max_retries):
        r = fn()
        if r.status_code != 429:
            return r
        sleep = min(2 ** attempt, 30) + random.uniform(0, 1)  # backoff + jitter
        time.sleep(sleep)
    raise RuntimeError("rate limited after retries")

2. Respect the headers. Many APIs return Retry-After or remaining-quota headers — honor them instead of guessing.

3. Smooth your traffic. Queue and pace requests to stay under RPM/TPM rather than firing in bursts. A token-bucket limiter on your side prevents most 429s.

4. Batch the non-urgent. Move anything a human isn't waiting on to a Batch API (also ~50% cheaper).

5. Trim tokens. Smaller prompts and capped output stretch your TPM further — the same discipline that cuts cost in how to reduce AI API costs.

How Gateways Change the Picture

A single provider key gives you that provider's limit. A gateway that pools capacity across providers can route around a saturated one and fail over automatically — so a spike that would 429 on one backend gets absorbed. If you're building on multiple models anyway, routing through LinkModel gives you one key across image, video, and chat models with a 99.95% SLA, and the same abstraction makes provider-level fallback trivial (try model A, catch, try model B). The fallback pattern is in build an AI app with multiple models.

# provider-agnostic fallback: if one model is saturated, try the next
for model in ("seedance-2-0", "kling-v3"):
    try:
        return generate(model, prompt)
    except RateLimited:
        continue

Checklist

• Implement exponential backoff with jitter on every call.
• Honor Retry-After / quota headers.
• Add a client-side rate limiter to smooth bursts.
• Batch non-urgent work; trim prompts and cap output.
• Design provider fallback so one saturated backend doesn't take you down.

Handled well, rate limits are a throughput-planning problem, not an outage. For the cost side of the same decisions, see the AI API pricing comparison.

Start free with a $1 credit and test your throughput across models on one key.