Get startedAPI referenceGoing to productionExplore models

Routing & reliability

Rate limits & retries

Request rate and concurrent requests are separate limits. A key can stay below its requests-per-minute limit and still exceed its concurrency limit while long streams remain open.

Identify the limit

429 codes
api_key_rate_limit_exceeded
api_key_concurrency_exceeded
model_rate_limit_exceeded
model_request_rate_limit_exceeded
model_request_success_rate_limit_exceeded

Retry with care

Honor Retry-After when present. Otherwise use bounded exponential backoff with jitter for 429 and transient 5xx failures. Do not repeatedly retry invalid input, expired keys, or exhausted balances.

A timeout can occur after the provider has processed your request. Retrying can repeat work and incur another charge. Set a retry budget, account for SDK retries, and never restart a stream silently after delivering partial output.

Copied