In brief
- Do not retry every non-success response.
- Use deadlines, jitter, and a strict attempt budget.
- Surface fallback and partial completion to callers.
Classify before recovering
Authentication, validation, rate limits, transient upstream failures, and timeouts require different responses. Retrying an invalid request wastes time; immediately failing a transient disconnect wastes availability.
| Class | Action | Retry |
|---|---|---|
| Authentication | Fix credential or permission | No |
| Validation | Fix request | No |
| Rate limit | Respect retry guidance | Bounded |
| Transient upstream | Backoff with jitter | Bounded |
| Client deadline | Cancel downstream work | Policy-dependent |
Use an end-to-end deadline
A per-attempt timeout is not enough. Give the whole operation a deadline, reserve time for response handling, and stop launching attempts when the remaining budget cannot support them.
Protect side effects
Text generation is often safe to repeat, but the workflow around it may create records, publish content, or call tools. Carry an operation ID and make side-effecting steps idempotent before enabling automatic retries.
Make recovery visible
Return structured error context, record attempt count and model identity, and explicitly mark fallback results. Silent recovery looks successful until a behavior difference reaches a user.
Reliability is not the absence of errors. It is bounded, explainable behavior when errors happen.
Frequently asked
Questions, answered plainly.
How many times should an AI API request retry?+
Use a small bounded attempt budget driven by the operation deadline and error class. There is no safe universal count.
Should rate-limit responses be retried?+
They may be retried when guidance and the remaining deadline permit it. Add jitter and avoid synchronized retry bursts.
Can I fall back to another model after a timeout?+
Yes, if the substitute supports the task and the behavior has been evaluated. Record that the fallback occurred.
Sources and next paths