Error Handling and Graceful Degradation

The Incident You Are Not Planning For

OpenAI had four major service disruptions in 2024. Anthropic has had rate limit events that lasted hours. Google Cloud has had AI inference outages. These are not edge cases — they are scheduled events on the calendar of any production AI product. The question is not whether your AI provider will fail. The question is what your users see when it does.

Most developers build the happy path and ship. The AI responds with valid data, the code processes it, the user gets what they asked for. That path works approximately 99% of the time. The 1% — the 429s, the 503s, the timeouts, the content refusals, the malformed outputs — that is where products get remembered for the wrong reasons.

This lesson is about building the other path. The path that runs when everything goes wrong.

The Error Taxonomy

Before you can handle errors, you need to know what you are handling. AI API errors fall into six categories, and they require different responses:

429 Rate Limit — you are sending requests faster than the provider allows, measured in requests per minute or tokens per minute. This is a traffic problem, not an API problem. The fix is to slow down.

503 Service Unavailable — the provider is down or overloaded. This is their problem, not yours, and it requires waiting. You cannot fix a provider outage by retrying faster — that makes it worse.

Timeout — the request exceeded your timeout setting. This might mean the provider is slow (their problem) or your timeout is too short (your problem). The response is the same: fail fast and serve the fallback.

Content Policy Refusal — the AI declined to answer. This is critically different from the above: the server responded successfully, but the model chose not to complete the request. This is NOT a 4xx or 5xx error. It arrives as a normal 200 response. On current Anthropic models, check the structured signal first: response.stop_reason === 'refusal' (with a stop_details object describing the refusal category) — that is the reliable detection path. Pattern-matching refusal phrases in the response text is fragile and should only be a fallback for older models or providers that lack a structured refusal signal. Either way, your code must detect refusals in the response payload, not in the HTTP status code.

Malformed Output — the AI returned a response, but it does not match the shape you expected. Valid JSON with the wrong fields. Plain text when you expected JSON. A numbered list when you expected a single paragraph. The server did its job; the model did not follow your format instructions.

Context Overflow — the input you sent exceeded the model's context window. This is a usage error on your side: you are sending more tokens than the model accepts. Fix: truncate input or use a model with a larger context window.

The Retry Strategy: Exponential Backoff

For server errors — specifically 429 and 503 — retrying is appropriate. But the timing matters. Retrying immediately after a 429 is the same as not catching the error at all. You are still sending at the rate that caused the problem.

Exponential backoff means each retry waits longer than the last: 1 second before retry 1, 2 seconds before retry 2, 4 seconds before retry 3. After three failed retries, stop. The provider is not coming back in the next few seconds.

The sequence:

attempt 1 (immediate) → 429 → wait 1s → attempt 2 → 429 → wait 2s → attempt 3 → still failing → fallback chain

For 503s, apply the same logic. If the service is down, retrying three times over 7 seconds is reasonable. Retrying 10 times over 40 seconds while the user stares at a spinner is not.

Timeouts do not benefit from retrying. If the call already took 15 seconds and failed, retry 1 will also take 15 seconds. Go directly to the fallback chain.

Handling Content Policy Refusals Correctly

This is the error type that most implementations get wrong. When the AI refuses to answer, developers often:

• Return the refusal message verbatim to the user ("I'm sorry, I can't help with that because...")
• Treat it as a server error and return a 500
• Retry the same request, which will fail the same way

The correct response is to detect the pattern, log it, and return a product-specific message that tells the user what they can do instead. "I can't help with that specific request. Here's what I can do..." is more useful than "I'm sorry, but I'm not able to assist with requests that..."

You also want to log content refusals separately. A spike in refusals often means your product is being used in a way you did not intend — which is safety-relevant information.

Circuit Breakers: Protecting Your Users During Outages

A circuit breaker watches the failure rate of AI calls over a rolling window. When failures exceed a threshold — say, 50% of calls failing in the last 60 seconds — the circuit opens. In the open state, requests bypass the AI entirely and go straight to the fallback chain. No network call is made. Users get an instant response instead of waiting 15 seconds for a timeout.

After the circuit has been open for 30 seconds, it transitions to HALF-OPEN: a small percentage of requests (10%) are allowed through to probe whether the provider has recovered. If those probe requests succeed, the circuit closes and normal operation resumes. If they fail, the circuit reopens and the 30-second cooldown restarts.

The 15-Second Rule

Set a 15-second timeout on every AI call. Not 30, not 60, not "let it run until it finishes." Fifteen seconds.

A user waiting 45 seconds for a response has already assumed the product is broken. They have likely refreshed the page, opened a new tab, or switched to a competitor. The response that arrives after 45 seconds gets returned to nobody.

The timeout is not negotiable because of what happens on the other side of it: your fallback chain. The user gets a real response — slightly degraded, perhaps from cache, perhaps static — instead of a blank screen. That is a better outcome than an accurate answer delivered too late.

In Knox's trading systems, every coroutine has a 30-second asyncio.wait_for timeout. An uncaught timeout in an order placement flow is not an inconvenience — it is a real money risk. The principle is identical: an unanswered request is worse than a gracefully declined one.

Handling Malformed Output

Your AI returns something, but it is not what you asked for. This happens more often than you would expect, particularly with complex format instructions.

The two-step response:

Step 1 — Re-prompt with explicit format requirements added. If your original prompt said "return the summary in JSON format," your re-prompt says "Return a JSON object with exactly these fields: title (string), summary (string, max 200 words), tags (array of strings, 3-5 items). No other text. Only the JSON object." This fixes the problem in most cases.

Step 2 — If the second attempt is also malformed, serve the static fallback. You have two data points now: the model is not following format instructions for this input. Do not retry a third time. Go to fallback and log the failure.

The thing you never do is silently fill in missing fields with defaults and continue. That hides a real signal — your prompt has a format reliability problem — and produces data your product treats as correct when it is not.

What This Looks Like in Production

Every AI call in a production system runs through this sequence:

1. Check circuit breaker — if open, go directly to fallback
2. Set 15-second timeout
3. Make the AI call
4. If 429/503: exponential backoff, retry up to 3 times, then fallback
5. If timeout: skip retries, go to fallback
6. If success: validate output shape
7. If malformed: re-prompt once, then fallback
8. If content refusal: detect pattern, return specific message
9. On fallback: check cache first, then serve static message

The fallback chain itself has a hierarchy: cached response (slightly stale but real data) beats static message (correct but generic). Users would rather see yesterday's answer than "our AI is temporarily unavailable" — but both are better than an error page.

Build this once, correctly, and it handles every failure mode you will encounter in production.