System Design · Lesson 42

Root Cause Analysis

The pager goes off at 2 AM. Checkout is down for one region. We fix it, we go back to bed, and three weeks later it happens again. That second outage is the one that costs us. Not because the system broke twice, but because the first time we never asked why it broke. A root cause analysis is the meeting where you ask that question properly. Not to find someone to blame. To find the guardrail that was missing, so the next person does not make the same mistake. This lesson walks the nine steps of an RCA I actually run, from the page to the follow-up. You will see the clean version, the messy version where the timeline has holes in it, and the version where nobody did the action items and the incident came back.

Difficulty: Intermediate · Know this first: Lesson 25, Lesson 01
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in minutes, not milliseconds. The bar below is a worked example for one RCA, not a measurement from a live incident, but the shape is honest: the timeline and the mitigation take the most time, and the steps that look quick on paper are the ones that decide whether the RCA changes anything. Step through the lesson or switch the scenario and watch the bar change. In the messy scenario, watch the timeline hop balloon to two hours. In the recurring scenario, watch the follow-up hop shrink to five minutes. That shrinkage is the whole story.

Turn the knobs and watch the next outage move

The budget above is one fixed RCA. This box is yours. Detect slower, mitigate slower, and leave half the action items undone. Then read the verdict. I run this math in my head after every RCA now, because the question is never whether the incident was bad. The question is whether it will happen again.

How long the problem ran before a human knew. It drives the detect hop and the total MTTR.

How long the users were down after detection. It drives the mitigate hop and the total MTTR.

How much of the last RCA's fix list actually got done. It drives the recurrence risk, not the budget total.

Total MTTR35 mindetect + mitigate
Recurrence risk20%undone items, adjusted up for a long outage
Action items completed80%of the last RCA's fix list

Formulas, stated plainly: total MTTR is time to detect plus time to mitigate. Recurrence risk starts as the share of action items left undone (100 minus completion), and it rises by 15 points when the MTTR runs over 60 minutes, and by another 10 when it runs over 120 minutes, capped at 100 percent. Verdict: under 25 percent risk is RCA healthy, 25 to 50 percent is RCA shaky, over 50 percent will recur. These are teaching numbers for one RCA, not measurements from a live incident.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Go deeper

The lesson above is one RCA, nine steps, three scenarios. Production is the rest of it. Here is what I actually spend my time on once the template is filled in and the meeting is over.

Blameless language, in practice

Blameless does not mean nobody did anything wrong. It means the document never names a person as the cause. I write "the migration removed the index" instead of "Priya removed the index". I write "the deploy went out without a check" instead of "the deploy was rushed by the on-call". The difference matters, because the moment a document names a person, the next person who makes a mistake will hide it instead of reporting it. I also watch the language in the meeting. "Why did you do that?" is a blame question. "What did the system let you do?" is a root cause question. Same moment, different question, different answer. I correct the language in the room, gently, every time, because the language is the culture.

Five whys done right, and where it goes wrong

Five whys works when every answer is a fact with evidence behind it. It goes wrong in three ways I see often. First, the chain stops too early, at the first technical cause, and the team fixes the query but never asks why the migration was untested. Second, the chain stops at a person, and the action item becomes "train the engineer", which fixes nothing, because the next engineer will make the same mistake. Third, the chain is built backwards from the answer the team already wanted, which is not analysis, it is a story. My check is simple: I read the chain from the bottom up. If the bottom answer does not explain the top impact, the chain has a gap. And I ask one more why after the fifth, just to check whether the fifth was really the end.

What makes an action item real

Four things, and I check all four before the meeting ends. What changes, stated so plainly that a new engineer could do it. Who owns it, one name, not a team. When it is due, a date, not "soon". And how we will know it is done, a test or a check, not a feeling. "Add a guardrail to the migration script" fails all four tests. "Priya adds a check that blocks a deploy when an index used by a production query is dropped, due in two weeks, verified by running the check against last month's migrations" passes all four. I also rank the items. The item that would have stopped this incident goes first. The item that makes the next incident easier to detect goes second. The rest can wait, and saying so out loud is part of the job.

Contributing factors vs root cause

I stopped saying "the root cause" in RCAs, and I say "the contributing factors" instead. The reason is simple: a real incident has a chain, and naming one link as the root cause lets the team fix that link and walk away feeling done, while the rest of the chain waits for the next incident. In this lesson's clean RCA, the factors were the slow query, the missing index, the untested migration, the slow alert, and the runbook gap. Cutting any one of them would have stopped the outage or made it shorter. The action items come from the whole list, not from one item. If your RCA has one root cause and one action item, you have not finished the analysis. You have finished the document.

What changes at 10x

At ten times the incidents, the RCA process breaks in a specific way: the meetings get shorter, the timelines get thinner, and the action items pile up faster than anyone can do them. I handle it by grading incidents. Not every incident gets a full nine-step RCA. A short outage with a clear cause gets a short RCA, written the same day. A long outage, a customer-facing outage, or a repeat gets the full treatment. I also cap the open action items. If the list grows past twenty, I stop writing new ones and ask which old ones can be closed or dropped, because an open list of fifty items is a list nobody believes in any more.

What changes at 100x

At a hundred times the incidents, the RCA stops being a document and becomes a system. I track the action items like a product backlog, with a weekly review and a burn-down. I tag every RCA with its contributing factors, so I can ask which factor shows up most often across all incidents, and that factor becomes the quarter's reliability project. And I measure the recurrence rate: how many incidents are repeats of an incident we already did an RCA for. If that number climbs, the problem is not the systems. It is the follow-up, and I say that plainly to the team.

RCA document template I actually use

This is the outline I start every RCA document from:

# RCA: [short title] — [date]

## Summary
One paragraph. What broke, for whom, for how long.

## Impact
Users affected, duration, severity. Numbers, not adjectives.

## Timeline
- 14:05 — Deploy went out (deploy log)
- 14:12 — First errors (metrics)
- 14:17 — Alert fired (paging system)
- 14:47 — Rollback finished (deploy log)
Every event has a time and a source. No memory.

## Five whys
1. Why did checkout fail? [fact, with evidence]
2. Why? [fact, with evidence]
3. Why? [fact, with evidence]
4. Why? [fact, with evidence]
5. Why? [fact, with evidence]

## Contributing factors
- [factor, as a system condition, not a name]
- [factor]
- [factor]

## Action items
- [ ] [what changes] — Owner: [one name] — Due: [date]
      Done when: [how we will know]
- [ ] [what changes] — Owner: [one name] — Due: [date]
      Done when: [how we will know]

## Follow-up
Weekly check until every item is done. Owner: [one name].

The template is the whole process in one page. Summary, impact, timeline, five whys, factors, action items, follow-up. If a section is empty, the RCA is not done. The follow-up section is the one most teams skip, and it is the one that decides whether the incident comes back.

One time it broke

This one stayed with me because the first RCA blamed a person, the incident came back, and the second RCA found the real cause sitting in plain sight the whole time.

What broke

A routine config change took down the checkout for one region for about an hour. The change was a small edit to a routing rule, made by hand in a console, and it sent payment traffic to a service that was not running. The first RCA was written the same day. It said the engineer who made the edit had not followed the change process, and the action item was to remind the team to follow the process. Three weeks later, the same outage happened again. A different engineer, a different console, the same mistake. The routing rule was edited by hand, the traffic went to a dead service, and checkout went down for the same region. The second RCA asked a different question. Not who made the edit. Why the system let a hand edit reach production at all.

Timeline

  • 09:40An engineer opens the routing console to adjust a rule for a planned change. The console has no confirmation step and no check against the live service list.
  • 09:43The edit is saved. Payment traffic for one region starts routing to a service that is not running. The error rate climbs.
  • 09:58The alert fires. The on-call engineer sees payment errors for one region and starts checking the payment service, which is healthy.
  • 10:15The team finds the routing change in the console history. The change is reversed by hand, the same way it was made.
  • 10:40Traffic recovers. The incident is declared over. The first RCA blames the engineer for not following the change process.
  • 13:20Three weeks later. A different engineer makes a different hand edit in the same console. Same result. Payment traffic goes to a dead service.
  • 13:35The alert fires. The team recognizes the pattern immediately and reverses the edit.
  • 14:05Traffic recovers. The second RCA asks why a hand edit can reach production at all.
  • +2 wksA guardrail ships: the console now checks every routing edit against the live service list, and it blocks any edit that points traffic at a service that is not running. Hand edits are also logged with the editor's name and the old and new values, so the timeline never depends on memory again.

What changed after

Three things, and I still do all three. First, the guardrail. A check that blocks a bad edit is worth more than a hundred reminders to be careful, because a check works when a person is tired, rushed, or new, and a reminder does not. Second, a rule for every RCA I run now: if the chain ends at a person, keep going. The person is never the last why. The system that let the person make the mistake is the last why, and that is where the action item lives. Third, I track repeats. Every RCA now asks whether this incident has happened before, and if it has, the first question is what happened to the last RCA's action items. An incident that repeats is not a new incident. It is an unfinished fix.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I run a real RCA. Take a minute with it before you read my approach.

The prompt

You are the incident lead for a 45-minute checkout outage. Payments failed for one region. The alert fired 20 minutes after the first failure, the rollback took 25 minutes, and nobody declared the incident until the second page went out. Write the RCA outline for this outage: the timeline, the five whys, the contributing factors, and the action items. What do you put in each section, and what do you leave out?

Clarifying questions to ask first

I would ask six things before I wrote a word. What time did the first payment fail, and where is that in the logs, not in anyone's memory? What changed in the hour before the first failure: a deploy, a config edit, a traffic shift? Why did the alert take 20 minutes: was the threshold wrong, was the alert noisy, or was nobody watching? Why did the rollback take 25 minutes: was there no runbook, or did the team try to diagnose first? Who declared the incident, and when, and why did it take a second page? And has this outage happened before, and if so, what happened to the last RCA's action items?

Rough shape of the answer, worked

Timeline first, from sources: first failure time from the logs, alert time from the paging system, declaration time from the channel, rollback start and finish from the deploy log. Then five whys, starting from the impact: payments failed for one region for 45 minutes. Why? Because the payment service returned errors. Why? Because its connection pool filled up, or the deploy broke the route, or the config pointed at a dead service, depending on what the evidence says. The chain ends at a process gap: a change with no guardrail, a test that was never run, or a runbook that was never written. Contributing factors: the slow alert, the late declaration, the slow rollback, and the cause itself. Four to five factors, each a system condition. Action items: three to five, each specific, owned, dated, verifiable. One item fixes the cause. One item fixes the detection. One item fixes the declaration or the rollback. One item checks whether this incident has happened before and what happened to the old action items.

Trade-offs to name

I spend the RCA's time on the timeline and the follow-up, not on the blame, because the timeline is what makes the rest of the analysis honest. I keep the action item list short, and I accept that some real problems will not make the list, because a list of twenty items is a list nobody finishes. I declare the incident early and stand it down fast, and I accept the occasional false alarm, because a late declaration costs more than an early one. And I publish the RCA even when it is uncomfortable, because an RCA that hides the slow alert or the late declaration is a document that lies to the next incident.

Next in this series
Lesson 43 — Disaster Recovery Setup
Coming next