Every AI feature I've shipped depends on infrastructure I don't control. An LLM provider outage isn't a design flaw you can engineer away, it's the deal you make when you call someone else's model. The flaw is shipping as if that dependency were as reliable as your own database.
It isn't. The first half of 2026 handed us multi-hour incidents at both of the providers most teams build on: a ten-hour Claude degradation in April, and OpenAI's API having its own bad day the same month. If your AI feature had no plan for those hours, your users got a spinner, then an error, then nothing.
"Add a fallback model" is the advice everyone gives and almost nobody implements properly. Here's what I actually do.
An LLM provider outage is rarely a clean "down"
Full outages are the easy case. The status page turns red, error rate goes to 100%, everyone knows what happened.
The expensive failures are the partial ones. 429s under load, only on your account, only at peak. p99 latency jumping from 4 seconds to 90 while p50 sits there looking fine. Valid JSON that's quietly worse because you got routed to degraded capacity. One region timing out while the status page still says "operational."
None of that trips a naive try/catch. And a request that hangs for 90 seconds does more damage than one that fails in 2, because it holds your connection, your serverless function's execution budget, and your user's attention all at once. So the first thing I add to a provider call isn't a fallback. It's an aggressive timeout, set from what the user experience can tolerate rather than whatever the vendor SDK ships with.
Retries aren't a fallback
Hammering the same provider during an incident is how you turn a ten-minute blip into a self-inflicted rate limit. If the failure is capacity, every retry makes the queue worse for you and for everyone else.
Three rules I hold to.
Retry only what's retryable. A 429 with a retry-after header, a 500, a connection reset: yes, with exponential backoff and jitter. A 400 because your prompt was malformed, or a content-policy refusal: never. Retrying those burns money and latency to arrive at the same answer.
Cap the wall clock, not the attempt count. "Three retries" means nothing if each attempt can take 60 seconds. Budget a deadline for the whole operation and check it before every attempt.
Trip a breaker. After N consecutive failures against a provider, stop calling it for a cooldown and send everything to the secondary. Teams skip this one, and it's the one that matters most: without it, every single request pays the full timeout before failing over. With it, only the first few do.
Keep the provider behind a seam
In every codebase I work in now, no feature code imports a vendor SDK directly. One module owns model selection, timeouts, retries, the breaker, and cost accounting. Feature code asks for a capability ("draft this post", "grade this answer"), never for a vendor.
That seam takes a couple of hours to build and it's worth doing on day one. Once it exists, swapping a model is a config change rather than a refactor. You can route by task, sending classification to something cheap and fast while the frontier model handles the thing users actually judge you on. Your fallback chain lives behind one call site instead of thirty. And your logs have one shape, so you can see which provider is misbehaving instead of guessing.
When I built the AI systems in Threadovo, which generates video scripts, ad copy and flyers against a credit budget, that seam is what let me change models without re-pricing every feature by hand. Same story in Scrivane, where the course-authoring AI routes through a gateway instead of a hardcoded vendor. I've written about retrofitting this kind of boundary into an existing codebase in How to Add AI to Your Existing Web App Without Rewriting It.
A gateway like OpenRouter gives you a lot of this out of the box and it's a reasonable default. Just remember you've now made the gateway your single point of failure, so keep one direct provider key configured as the escape hatch.
Decide the degradation ladder per feature
This part is a product decision, not an engineering one. Falling back to another model isn't always the right answer.
For each AI surface I write down what happens at each rung:
- Primary model. The normal path.
- Fallback model. Different provider, prompt already validated against it. A fallback you've never evaluated is just a different kind of outage.
- Cheaper or older model. Worse output, still useful.
- No AI at all. Templates, cached results, last known good output, a queue.
- Honest failure. Tell the user, don't charge them, let them retry.
The ladder differs per feature. For a background job like generating a weekly content plan, rung 4 is "retry in ten minutes and email them when it's ready," and nobody notices. For something interactive, rung 4 might be showing the previous result with a "generated earlier" label.
The mistake is defaulting every feature to rung 2. Quietly swapping to a weaker model on a surface where quality is the product is worse than an error message, because now you've shipped bad output under your own name and the user has no idea why.
Voice doesn't get a ladder
Real-time voice is the exception. I learned that building the voice pipelines in Intavue and the citizen-call assistant in AmtHeld. In a text UI a two-second retry is invisible. On a phone call, two seconds of silence is a person going "hello? are you there?"
There's no time to fail over mid-turn, so the design inverts: you race instead of retrying. Once the primary misses its latency budget, fire the fallback in parallel, take whichever answers first, cancel the other. You pay for two calls occasionally. Much cheaper than a call that dies.
It also helps to have a filler ready, an acknowledgement or a "let me check that," to buy 800ms of cover without lying to the caller.
Don't bill for work you didn't deliver
If you meter AI usage, failure has an accounting side. Reserve credits when the job starts, settle on success, release on failure. If you fell back to a cheaper model, charge the cheaper rate or eat the difference. Don't charge frontier prices for a fallback the user never asked for.
I went into the mechanics in Credit-Based Billing for AI Features. Short version: a refund your support inbox has to process by hand costs more than the credits did.
Instrument it before you need it
You can't fail over from something you can't see. Per provider and per model I want request count, error rate split by class (429 / 5xx / timeout / refusal), p50 and p95 latency, fallback rate, and cost.
Fallback rate is the leading indicator. When it drifts from 0.2% to 8%, something upstream is wrong, usually well before the status page admits it. Subscribe to the provider status feeds (Anthropic publishes one) but don't treat them as your detection layer. You'll know first.
Then test the thing. A fallback path that has never run in production isn't a fallback, it's a hypothesis. Force it on purpose: flip the primary to an invalid key in staging, or better, shadow a small share of production traffic to the fallback so you know the prompt still holds up over there.
If you only do three things
Timeouts tied to the user experience. A circuit breaker, so you fail over fast instead of one request at a time. One provider seam so all of it lives in a single file.
That's about a day of work, and it decides whether an LLM provider outage is something your users never notice or a morning spent writing refund emails.