System Design · Lesson 52

Chaos Engineering

I break my own systems on purpose, in daylight, with the team watching. That is chaos engineering. Not random destruction. A written hypothesis, a small blast radius, and a kill switch I tested before the fault goes in. This lesson runs one controlled experiment end to end, the way I run it on a game day.

Difficulty: Advanced · Know this first: Lesson 22, Lesson 43

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every step of an experiment costs minutes. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most experiments I have run, the write-up and the watching cost more than the fault itself. Step through the lesson or switch the scenario and watch the bar change.

Try it: turn the fault up and watch the system answer

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the error rate, the detection time, and the resilience score move, and watch the watch step, the user impact step, and the stop step change in the budget bar. The numbers are a teaching example, not live measurements. The shape is the lesson: a bigger fault is easier to notice and harder on users, and the score punishes user pain, not detection speed.

10

Share of the checkout pods the fault kills. At 0 percent nothing is injected. At 50 percent half the pods are gone at once. Bigger faults are detected faster and hurt more.

5

Share of user traffic inside the experiment. At 1 percent almost nobody can be hurt. At 25 percent a quarter of users are in the blast radius. The error rate grows with fault times blast radius.

5

The error rate that pages the on-call. A low threshold pages fast and pages often, including on noise. A high threshold stays quiet longer and lets real damage grow before anyone is called.

Try this: (1) At the defaults, 10 percent fault, 5 percent blast radius, 5 percent threshold, the error rate is 1.2 percent, detection takes 105 seconds, the resilience score is 96, and the experiment totals about 56 minutes. (2) Turn it all the way up: 50 percent fault, 25 percent blast radius, 10 percent threshold. The error rate is 25.2 percent, the score falls to 24, and the experiment totals about 84 minutes, most of it in user impact. Same system, same code. Only the inputs changed. (3) Turn the fault to 0 and the blast radius to 1: the error rate is 0.2 percent and the score is 99, but the experiment proved nothing, because nothing was injected.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Go deeper

A game day is the full version of this lesson. Once a quarter, the whole team gathers in business hours with the dashboards open, and one person runs one fault while the rest watch. The rules are simple: the fault is announced, the blast radius is written down, the kill switch is tested first, and someone owns the write-up before the day ends. A game day that finds nothing is still a good day. It means the system held, and the team practiced the motions they will need at 2 AM.

Chaos in CI is the smaller, cheaper version. A pipeline step that kills a dependency in a test environment on every merge, or adds 500 ms of latency to one downstream call, catches brittle retries and missing timeouts before production ever sees them. The fault is small and the blast radius is zero users. I would rather learn in CI that a service retries forever than learn it from a real outage.

Production versus staging is the honest trade-off. Staging lies about traffic shape, data size, and real user behavior. An experiment that only runs in staging tests the staging system. Production experiments prove real resilience, but they need guardrails staging does not: business hours, a small radius, a tested kill switch, a named person watching, and a hard stop if the error budget burns too fast. I try the fault in staging first, then I prove it in production at a radius I can defend to the business.

Measuring resilience is how the program earns its keep. I track the same four numbers after every experiment: detection time, peak error rate, time to recover, and whether the hypothesis passed. Then I trend them quarter to quarter. If the score never moves, the experiments are too easy. If every experiment fails, the radius is too big or the system is not ready. A chaos program with no trend line is a hobby. A trend line that improves is evidence the system is getting stronger.

At 10x the traffic, a 1 percent blast radius is ten times the users. The radius shrinks as traffic grows, and the guardrails get tighter. At 100x, experiments run per region or per cell, never across the whole fleet at once, and the kill switch gets its own owner and its own test. The experiment design stays the same. Only the numbers get smaller and the stakes get bigger.

# chaos experiment: kill pods in the checkout service
# one fault, a small radius, a tested kill switch
apiVersion: chaos.example/v1
kind: PodKillExperiment
metadata:
  name: checkout-pod-kill
spec:
  target:
    service: checkout
    namespace: prod
    selector:
      app: checkout
  fault:
    type: pod-kill
    percent: 10
    duration: 5m
  guardrails:
    maxErrorRate: 1%
    abortOn:
      - errorRateAbove: 2%
      - p99Above: 800ms
    killSwitch: chaos-abort-checkout
  hypothesis:
    errorRateStaysUnder: 1%
    p99StaysUnder: 400ms
  steadyState:
    metric: checkout_success_rate
    window: 10m

What the snippet means: the fault kills 10 percent of the checkout pods for five minutes, and nothing else. The guardrails abort the experiment automatically if the error rate passes 2 percent or p99 passes 800 ms, and the kill switch is named so a human can stop it by hand. The hypothesis is written into the experiment itself, so pass or fail is a fact, not an opinion. And the steady state window says what normal looked like before the fault, so the result has something to compare against.

One time it broke

What broke

During a game day, the fault was supposed to kill pods in the payment service. It did. The problem was the abort job. The abort ran as a pod inside the same service, on the same set of pods the fault was killing. When the on-call hit the kill switch, the job that was supposed to stop the fault was already dead. The experiment could not stop itself, and the team had to stop it by hand while the error rate kept climbing.

Timeline

  • 10:00,the fault starts. Ten percent of the payment pods are killed, as planned.
  • 10:02,the error rate climbs past 2 percent. The alert fires and pages the on-call.
  • 10:04,the on-call hits the kill switch. Nothing happens. The abort job does not run.
  • 10:06,someone checks and finds the abort job is a pod in the same service. The fault killed its own off switch.
  • 10:09,an engineer stops the fault job by hand from a laptop, outside the cluster.
  • 10:14,the error rate is back to baseline. The experiment is over, five minutes later than it should have been.

What changed after

Three things. First, the abort now runs outside the blast radius, in its own namespace on its own nodes, and it is tested before every experiment, not just reviewed. Second, every fault job gets a hard time limit, so it dies on its own even if the abort fails. A fault that cannot stop itself is not a controlled experiment. Third, the label selector on every experiment gets a second pair of eyes before the run. The selector was right that day, but the abort placement was wrong, and a review that only checks the fault misses half the risk.

If you remember nothing else

Design exercise

Design a chaos program for a checkout service that runs 500 orders a minute. You can run experiments in production twice a month. The business will not accept more than 0.5 percent failed checkouts during an experiment. You have pod kill, latency injection, and dependency failure as fault types.

Clarifying questions to ask

  • What does 0.5 percent mean in orders: 2.5 failed orders a minute, and for how many minutes?
  • Which fault type first: pod kill, latency, or a dead dependency, and what does each one prove?
  • Who watches the run, who can hit the kill switch, and what happens if that person is away?
  • What is the abort line: error rate, p99, or failed orders per minute, and who set the number?
  • Where does the write-up live, and who owns the fixes it finds?

Rough sizing

Start with the budget. 0.5 percent of 500 orders a minute is 2.5 failed orders a minute, so the blast radius has to stay small. At a 1 percent blast radius, 5 orders a minute are inside the experiment, and if every one of them failed I would already be over budget. So I start at a 0.5 percent radius, 2 to 3 orders a minute, and I start with latency injection, not pod kill, because latency slows requests while a kill can fail them outright. I run one fault per session, five minutes of fault, ten minutes of steady state before it, and the whole run fits in one hour including the write-up.

Trade-offs to name

A small radius buys safety and costs learning speed. Production buys truth and costs risk, so the radius stays small and the runs stay in business hours. Latency faults buy safety and cost realism, so I rotate in a pod kill once the latency runs pass. Business hours buy a fast response and cost realism about night failures, so I note that gap in the write-up instead of pretending it does not exist. And two runs a month is a pace, not a target. If a run finds a real weakness, the next run waits until the fix ships.

Next in this series
Lesson 53 - Incident Management & On-Call
Coming next