Handle AI rate limits with a queue and bounded retries
Read the actual error and limit dimension, pace concurrent work and avoid retry storms that make a temporary limit worse.
In brief
An API limit can apply to request count, token volume or another resource. A 429 response needs its actual error context: a temporary rate limit and exhausted quota are not resolved by the same action.
A bounded queue can keep bursts from overwhelming the provider or your own server.
Identify the limiting dimension
Check the project/model limits and request size. A few very long prompts can exhaust token capacity even when request count is low. Do not rotate accounts or keys to evade policy; size and schedule legitimate work to its authorized capacity.
Coordinate retries in one place
Know whether the SDK already retries before adding a second retry loop. Bound attempts and total wait time, and spread retries rather than having every client retry immediately together. Retry authentication or invalid-input errors only after correcting their cause.
Keep state and users informed
For a queued task, display a truthful waiting or failed state. For a write tool, reconcile whether the first attempt acted before repeating it. For a large independent workload, consider a supported batch path when its deadline and limits fit.
Before you act
Failed requests and automatic SDK retries can still affect usage and timing. Unlimited retries can amplify a problem and duplicate consequential tool work.
What to do next
- Inspect the status, provider error and non-secret request reference.
- Use a queue or concurrency limit and obey Retry-After where provided.
- Apply bounded backoff with jitter for retryable errors, separating quota and authentication problems.
