I break my own systems on purpose, in daylight, with the team watching. That is chaos engineering. Not random destruction. A written hypothesis, a small blast radius, and a kill switch I tested before the fault goes in. This lesson runs one controlled experiment end to end, the way I run it on a game day.
Difficulty: Advanced · Know this first: Lesson 22, Lesson 43
Every step of an experiment costs minutes. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most experiments I have run, the write-up and the watching cost more than the fault itself. Step through the lesson or switch the scenario and watch the bar change.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the error rate, the detection time, and the resilience score move, and watch the watch step, the user impact step, and the stop step change in the budget bar. The numbers are a teaching example, not live measurements. The shape is the lesson: a bigger fault is easier to notice and harder on users, and the score punishes user pain, not detection speed.
Share of the checkout pods the fault kills. At 0 percent nothing is injected. At 50 percent half the pods are gone at once. Bigger faults are detected faster and hurt more.
Share of user traffic inside the experiment. At 1 percent almost nobody can be hurt. At 25 percent a quarter of users are in the blast radius. The error rate grows with fault times blast radius.
The error rate that pages the on-call. A low threshold pages fast and pages often, including on noise. A high threshold stays quiet longer and lets real damage grow before anyone is called.
Try this: (1) At the defaults, 10 percent fault, 5 percent blast radius, 5 percent threshold, the error rate is 1.2 percent, detection takes 105 seconds, the resilience score is 96, and the experiment totals about 56 minutes. (2) Turn it all the way up: 50 percent fault, 25 percent blast radius, 10 percent threshold. The error rate is 25.2 percent, the score falls to 24, and the experiment totals about 84 minutes, most of it in user impact. Same system, same code. Only the inputs changed. (3) Turn the fault to 0 and the blast radius to 1: the error rate is 0.2 percent and the score is 99, but the experiment proved nothing, because nothing was injected.
Three questions. No score is saved anywhere. This is just you against the material.
A game day is the full version of this lesson. Once a quarter, the whole team gathers in business hours with the dashboards open, and one person runs one fault while the rest watch. The rules are simple: the fault is announced, the blast radius is written down, the kill switch is tested first, and someone owns the write-up before the day ends. A game day that finds nothing is still a good day. It means the system held, and the team practiced the motions they will need at 2 AM.
Chaos in CI is the smaller, cheaper version. A pipeline step that kills a dependency in a test environment on every merge, or adds 500 ms of latency to one downstream call, catches brittle retries and missing timeouts before production ever sees them. The fault is small and the blast radius is zero users. I would rather learn in CI that a service retries forever than learn it from a real outage.
Production versus staging is the honest trade-off. Staging lies about traffic shape, data size, and real user behavior. An experiment that only runs in staging tests the staging system. Production experiments prove real resilience, but they need guardrails staging does not: business hours, a small radius, a tested kill switch, a named person watching, and a hard stop if the error budget burns too fast. I try the fault in staging first, then I prove it in production at a radius I can defend to the business.
Measuring resilience is how the program earns its keep. I track the same four numbers after every experiment: detection time, peak error rate, time to recover, and whether the hypothesis passed. Then I trend them quarter to quarter. If the score never moves, the experiments are too easy. If every experiment fails, the radius is too big or the system is not ready. A chaos program with no trend line is a hobby. A trend line that improves is evidence the system is getting stronger.
At 10x the traffic, a 1 percent blast radius is ten times the users. The radius shrinks as traffic grows, and the guardrails get tighter. At 100x, experiments run per region or per cell, never across the whole fleet at once, and the kill switch gets its own owner and its own test. The experiment design stays the same. Only the numbers get smaller and the stakes get bigger.
# chaos experiment: kill pods in the checkout service
# one fault, a small radius, a tested kill switch
apiVersion: chaos.example/v1
kind: PodKillExperiment
metadata:
name: checkout-pod-kill
spec:
target:
service: checkout
namespace: prod
selector:
app: checkout
fault:
type: pod-kill
percent: 10
duration: 5m
guardrails:
maxErrorRate: 1%
abortOn:
- errorRateAbove: 2%
- p99Above: 800ms
killSwitch: chaos-abort-checkout
hypothesis:
errorRateStaysUnder: 1%
p99StaysUnder: 400ms
steadyState:
metric: checkout_success_rate
window: 10m
What the snippet means: the fault kills 10 percent of the checkout pods for five minutes, and nothing else. The guardrails abort the experiment automatically if the error rate passes 2 percent or p99 passes 800 ms, and the kill switch is named so a human can stop it by hand. The hypothesis is written into the experiment itself, so pass or fail is a fact, not an opinion. And the steady state window says what normal looked like before the fault, so the result has something to compare against.
During a game day, the fault was supposed to kill pods in the payment service. It did. The problem was the abort job. The abort ran as a pod inside the same service, on the same set of pods the fault was killing. When the on-call hit the kill switch, the job that was supposed to stop the fault was already dead. The experiment could not stop itself, and the team had to stop it by hand while the error rate kept climbing.
Three things. First, the abort now runs outside the blast radius, in its own namespace on its own nodes, and it is tested before every experiment, not just reviewed. Second, every fault job gets a hard time limit, so it dies on its own even if the abort fails. A fault that cannot stop itself is not a controlled experiment. Third, the label selector on every experiment gets a second pair of eyes before the run. The selector was right that day, but the abort placement was wrong, and a review that only checks the fault misses half the risk.
Design a chaos program for a checkout service that runs 500 orders a minute. You can run experiments in production twice a month. The business will not accept more than 0.5 percent failed checkouts during an experiment. You have pod kill, latency injection, and dependency failure as fault types.
Start with the budget. 0.5 percent of 500 orders a minute is 2.5 failed orders a minute, so the blast radius has to stay small. At a 1 percent blast radius, 5 orders a minute are inside the experiment, and if every one of them failed I would already be over budget. So I start at a 0.5 percent radius, 2 to 3 orders a minute, and I start with latency injection, not pod kill, because latency slows requests while a kill can fail them outright. I run one fault per session, five minutes of fault, ten minutes of steady state before it, and the whole run fits in one hour including the write-up.
A small radius buys safety and costs learning speed. Production buys truth and costs risk, so the radius stays small and the runs stay in business hours. Latency faults buy safety and cost realism, so I rotate in a pod kill once the latency runs pass. Business hours buy a fast response and cost realism about night failures, so I note that gap in the write-up instead of pretending it does not exist. And two runs a month is a pace, not a target. If a run finds a real weakness, the next run waits until the fix ships.