Part V · Landing it · Chapter 15
When it breaks
Ten ways AI systems fail in production, most of them not hallucination, and a trace-first way to debug each one.
15 min read
Every deployment eventually has a bad day. What separates a good FDE is not avoiding incidents but handling them: limiting the damage quickly, finding the real cause rather than the convenient one, and making sure the same thing can't happen again quietly.
What you'll take away
- 01Why 'the model hallucinated' is usually the wrong diagnosis.
- 02Eight things users say when something is wrong, and where each one points.
- 03What to do in the first hour, before you know the cause.
- 04How to debug from a single trace instead of re-running questions in the UI.
- 05How to turn an incident into a permanent invariant, a test and a dashboard panel.
Most failures aren't hallucination
When an AI system gives a wrong answer, the instinct is to blame the model and edit the prompt. In production, the model is often the one component that behaved correctly. It faithfully used stale data, a document it should never have seen, or a citation the retrieval step mixed up. Fixing the prompt in those cases changes nothing, or hides the real problem until it returns somewhere worse.
Before touching the prompt, prove the model saw what it should have seen.
Start from what people report
Incidents rarely arrive labelled. They arrive as a sentence from a user or a manager. Each sentence tends to point to a different part of the system, so learn to translate them.
| What you hear | Common causes | First thing to check |
|---|---|---|
| "The answer is out of date" | The copy the system searches lags behind the original; a sync job is failing for part of the data | When the cited source last changed versus when the system last read it |
| "It says it can't find it, but it's there" | The document failed to load properly, or a filter excludes it for this user | Whether the content was stored at all, and the exact filter used for this request |
| "I'm seeing someone else's information" | A filter or access check skipped; a cached answer reused for the wrong person | Who owns each item the system used, compared with who asked |
| "It works for them but not for me" | A setting, region, feature flag or release that differs between users | What is different about the two users' requests |
| "It did it twice" | A timed-out action retried without protection; two steps run in the wrong order | Every tool call and its result in that conversation |
| "It's been worse since Tuesday" | A new model, prompt, index or setting | Everything deployed or changed that day, on both sides |
| "It's slowly got worse" | The questions or data have shifted: new products, new jargon, new document types | A sample of recent inputs against the test set |
| "The bill doubled" | Calls per unit went up, prompts grew, or saved results stopped being reused | Calls and tokens per unit of work, by day |
The first hour
In the first hour you are not looking for the root cause. You are limiting the damage and preserving the evidence.
- 1
Judge the severity
Is data exposed, money moving, people harmed or misled, or a regulation involved? Those are critical, and the people on your escalation list need to know now. - 2
Contain the damage
Roll back the last change, flip the feature switch, block the failing path, or route the affected case type to people. A reversible step now beats a perfect fix later. - 3
Preserve evidence
Save traces, versions and the exact inputs before caches expire or logs rotate. - 4
Find the blast radius
Which users, tenants, time window and versions are affected? Query the logs for the pattern, not just the one reported case. - 5
Communicate
A short, factual update to the client contact: what happened, what you've done, what happens next, and when they'll hear from you again.
Debug from the trace, not the UI
Re-typing the question in the interface is the most common first move and among the least useful. The cache may answer differently, the user's permissions differ from yours, and the deploy may have changed since. Start from the stored trace of the bad response and walk it step by step.
Compare, don't just inspect
A single trace tells you what happened. A comparison tells you why. Compare the bad trace with a good one for a similar request, the new version with the old, the users on the latest release with the rest, and peak hours with quiet ones. Then ask the oldest question in operations: what changed recently? A deploy, a configuration change, a provider update or a reorganised folder on the client's side explains most incidents.
From mitigation to a permanent fix
A mitigation stops the bleeding. A permanent fix makes the failure impossible, or at least loud. Each serious incident should leave behind four things:
| Leave behind | Example |
|---|---|
| An invariant in code | Every retrieved item's tenant must match the request's tenant, or the response is blocked |
| A test that would have caught it | An eval case or a deploy check that fails on cross-tenant retrieval |
| A signal on a dashboard | A panel counting invariant violations, by version |
| A runbook entry | How to recognise this failure next time and the first three steps |
The incident review
Write a short review for every significant incident, in the blameless style Google's SRE teams describe: focus on how the system allowed the failure, not on who made the change. Include a timeline, the impact, the root cause, what went well, what didn't, and actions with owners and dates. Share a version with the client. Clients forgive incidents. They don't forgive finding out you hid one.
Try it
It works for some branches
A building society rolls out an assistant that answers staff questions about mortgage products. Two weeks in, staff in the Scottish branches say it keeps quoting fees that don't apply to them. Staff in England report no problems. Here are two stored traces for the same question, asked by staff in Glasgow and in Leeds.
leeds user_region=ENG filter="region in (ENG, ALL)" sources=[fees_eng_v7, products_all_v3]
glasgow user_region=SCO filter="region in (SCO, ALL)" sources=[products_all_v3]
documents for SCO: fees_sco_v7 (status=failed_to_load, 14 days ago)Explain what happened, what you would do today, and what you would change so it can't happen silently again.
Write your answer down first, then reveal the worked answerWorked answer
What happened: the Scottish fee document failed to load two weeks ago, so Scottish staff were only ever shown the general product document. The model filled the gap with fees from that document, which reflect the English schedule. Filters and permissions worked correctly; the data simply wasn't there.
Today: tell Scottish branches to stop relying on fee answers, reload the document and confirm it is searchable, then check a sample of the past two weeks' Scottish fee answers to see who may have been misinformed.
So it can't happen silently: treat a failed load of any current document as an alert, not a log line; keep a short list of known questions per region that must return the right document, and run it daily; and have the assistant say it has no fee information for a region rather than answering from a general document. A strong answer notices the problem looked like a model error but was a missing file.
Go deeper
Practise the skills
- Reliability & incident responseBuild systems that fail safely, and stay calm when they break at 2 AM.
- Security, identity & data governanceEarn the CISO's yes: auth, permissions, tenancy and data handling done properly.
- Evaluation & qualityProve it works before launch, and know the moment it stops working after.