In brief
- Retry and fallback solve different failures.
- Capability compatibility is the first fallback gate.
- Graceful degradation can be safer than hidden substitution.
Four recovery modes
A transient retry repeats the same contract. A model fallback changes the implementation. Graceful degradation narrows the result. Queueing trades immediacy for eventual completion. Choose deliberately rather than chaining all four.
| Mode | Best fit | Primary risk |
|---|---|---|
| Retry | Brief transient failure | Traffic amplification |
| Model fallback | Primary unavailable | Behavior drift |
| Degrade | Optional enrichment | Reduced utility |
| Queue | Non-interactive work | Stale results |
Gate on required capability
Before substituting a model, confirm context size, modalities, tool support, structured output requirements, and regional or policy constraints. Price and availability are secondary if the model cannot fulfill the contract.
Test fallback quality independently
Run the fallback on the same evaluation set and inspect the cases where it diverges. A substitute may be acceptable for summaries and unsafe for tool decisions, even when both requests return 200.
Keep the path visible
Attach fallback metadata to traces and responses where appropriate. Alert on sustained fallback rate: a successful substitute can otherwise hide a degraded primary service for hours.
Frequently asked
Questions, answered plainly.
Is a retry the same as a fallback?+
No. A retry repeats an operation, usually against the same contract. A model fallback changes the model and potentially its behavior.
Should users know when a fallback occurs?+
Expose it when model identity or behavior is part of the product contract. At minimum, operators must be able to trace it.
Can a smaller model be a fallback?+
Yes for tasks where it has passed the required quality and capability gates. Degrade explicitly when it cannot provide the full result.
Sources and next paths