Rate limits, concurrency and queueing for production image pipelines

The difference between a demo and a production image pipeline is what happens at the 95th percentile of load. This post covers the two limits you'll meet, the client architecture that handles them gracefully, and the capacity math to know your real throughput.

The two limits#

1. Per-key RPS (default 10 req/s): how fast you may talk to the API. Exceeding it returns 429 rate_limited with Retry-After.
2. Per-account max in-flight (default 20): how many jobs may be queued or running at once. Exceeding it returns 429 too_many_inflight. GET /v1/account reports both inflight and max_inflight, so you can see headroom in real time.

RPS protects the front door; in-flight protects fairness of the generation queue. Both are raisable — ask from the dashboard when your steady-state needs it.

Client architecture that doesn't melt#

Push work through your own bounded queue rather than firing user requests straight at the API:

user requests → your queue → N workers (N ≤ max_inflight) → submit → webhook/poll → deliver

• Size N from the limit, not from hope. With max_inflight = 20 and ~8 s median generation, your ceiling is ~2.5 images/s sustained (20 ÷ 8). Want 10/s? That's a limit increase conversation, not a retry loop.
• Treat 429 as backpressure, not an error. Sleep Retry-After, don't count it toward user-visible failures, and never retry it with a fresh Idempotency-Key (that's how duplicate-charge bugs are born elsewhere; here it just wastes a queue slot).
• Prioritise interactively. One queue for user-facing requests, one for batch/backfill; drain batch only when inflight < max_inflight × 0.7.

Spikes, failures, and the death spiral you avoid#

Classic failure mode: a spike → timeouts → clients retry → the retry traffic doubles the spike. Two properties here break the spiral:

• Submits are cheap and answered in milliseconds even when the generation queue is deep — your requests don't time out, they queue.
• Retried submits with the same Idempotency-Key are free no-ops, so aggressive retry policies can't amplify load into double work.

Add jittered exponential backoff on transport errors (0.5 s → ×1.6 → cap 5 s) and you're done — no circuit breakers needed for the API path.

Monitoring that matters#

Watch three numbers: your queue depth, inflight / max_inflight, and the job failure rate by error.code. Failure spikes cost you nothing in credits (auto-refunds), but they're your earliest signal of bad prompts or upstream trouble. Everything else — balances, per-job costs, ledgers — is already reconciled server-side (how).

Reference: rate limits · account endpoint · webhooks.