Reliability & incident response
Build systems that fail safely, and stay calm when they break at 2 AM.
Roadmap: Production GenAI & integration, Client simulation & ownership
Why it matters
"Production-ready" means it handles partial failure, you can see inside it, and you can roll it back. When something does break at a customer site, the FDE is usually first on the call. How you handle those first 90 minutes decides whether the customer trusts you more or less afterwards.
What it's made of
- Timeouts, retries with backoff and jitter, idempotency keys
- Telling permanent failures from transient ones
- Dead-letter queues, replay and reconciliation
- Logs, metrics and traces tied together by request ID
- Feature flags, rollback and backfill plans
- Severity levels described in customer terms
- Incident command, customer updates and postmortems
Design for partial failure
Assume every network call can hang, fail or succeed without telling you. Put a timeout on everything. Retry only transient errors, with exponential backoff and jitter, and a cap. Make writes idempotent so a retry can't double-charge or double-send. When an external action's outcome is unknown, check its state before retrying.
The first 90 minutes of an incident
When the customer calls at 2 AM, work in this order:
- Minutes 0–5: acknowledge, confirm impact in their words ("recruiters can't upload CVs"), set a time for the next update.
- Minutes 5–20: stop the bleeding with reversible moves: roll back, flip a flag, pause write tools, isolate a tenant.
- Minutes 20–60: preserve evidence (logs, request IDs, versions, tool calls) and find the cause. One person investigates, one communicates.
- Minutes 60–90: confirm recovery by re-running the real user journey, then send a clear update on what happened and what's next.
- Afterwards: a blameless postmortem with owners and dates, shared with the customer.
Principles to hold under pressure
- Stabilise first, diagnose second.
- Prefer reversible actions.
- Say what you don't know yet, and when you'll know more.
- Evidence over ego: follow the data, even when it points at your code.
- Keep one source of truth for status, so the customer hears one story.
Mid-level looks like
Adds logs, alerts and retries, and can find a bug once someone points at it.
Senior looks like
Designs fallback chains and launch checklists up front, protects partial progress, and leads an incident calmly with regular, honest updates.
Common pitfalls
- Retrying forever, turning a small outage into a self-inflicted one.
- Writing a hotfix under pressure when a one-click rollback was available.
- Declaring recovery because the graphs are green, without re-running the user's actual journey.
- Blaming the vendor or the customer's data in the middle of an incident.
Practice drills
Drill 1
It's 2 AM and an AI agent is sending duplicate emails to a customer's clients. Talk through your first 10 minutes and write your first customer update.
A strong answer: Pausing the send tool immediately, preserving evidence, a calm four-sentence update with impact, action taken and next update time.
Drill 2
Build a failure register for an email-sending workflow where the send succeeds but the callback times out.
A strong answer: Cause, detection, mitigation and owner for each failure; idempotency keys; reconciliation before retry.
Drill 3
Write a four-sentence incident update for a non-technical executive.
A strong answer: Impact in business terms, what's been done, what's still unknown, and when they'll hear next. No jargon, no blame.
Read next
Designing Data-Intensive Applications
Martin Kleppmann
The best single explanation of how databases, replication, partitioning, streams and batch processing really work. You will meet every one of these in customer environments, and this lets you reason about them quickly.
Fundamentals of Data Engineering
Joe Reis, Matt Housley
Covers the full data lifecycle (ingestion, storage, transformation, serving) in vendor-neutral terms. Most FDE deployments start with getting at messy customer data, and this gives you the vocabulary and patterns for it.
Release It!
Michael T. Nygard
Stability patterns and anti-patterns from real production failures: timeouts, circuit breakers, bulkheads, back-pressure. Read it before you ship an integration into infrastructure you do not control.
Fundamentals of Software Architecture
Mark Richards, Neal Ford
Catalogues architecture styles and, more importantly, how to analyse tradeoffs between them. Helps you justify design choices to customer architects and review boards with more than personal preference.