A design that only describes the happy path is half a design. The other half answers the questions an SRE asks before go-live: what fails, how often, what the customer sees when it does, how we know, who gets woken, and how we change this thing without breaking it. Reliability is cheapest when it is designed in, as capacity headroom, failure domains, degraded modes, SLOs and alerts, and a rollout plan, and most expensive when it is discovered in the first incident. This lesson follows one tracking request while an entire availability zone is down, and shows the design decision that keeps each hop working. By the end you can write the reliability section of a design: the failure-mode table, the SLOs and their alerts, the capacity plan, and the rollout.
Difficulty: Master · Know this first: Lesson 169, Lesson 144, Lesson 43, Lesson 61
This bar splits one request during a zone outage. It is a worked example, not a measurement. When the design planned for the outage, the bar barely changes from a normal day; when it did not, one hop, capacity or a hidden dependency, becomes the whole bar.
Move the sliders. Lose one zone and its traffic moves to the others. Watch the surviving zones' utilisation, and the API hop in the bar above.
utilisation after losing one zone = peak utilisation × zones ÷ (zones − 1) · comfortable below 80 percent; over 100 percent the survivors cannot carry the load.
error budget per 30 days = (1 − SLO) × 43,200 minutes · N+1 peak target = 80% × (zones − 1) ÷ zones
N+1 holds: the survivors carry the peak
Three questions. No score is saved anywhere — this is just you against the material.
Designing for reliability grows in four levels: first naming how each part fails, then SLOs, capacity, and degraded modes, then blast radius and critical dependencies, and finally changing a running system safely and proving the design by breaking it on purpose.
At the basic level you add a failure-mode table to the design: one row per component, with how it fails, what the customer sees, and what takes over. You find single points of failure, the boxes with no row in the "what takes over" column, and you fix them or write down why they are acceptable.
Types:
Uses: every design review and every launch checklist.
Where it breaks: a table with only crash rows. Slowness and wrong answers cause most real incidents, because nothing restarts and nothing alerts.
Try it: Basic
Which failure is usually the hardest for a design to handle?
Now do it: Add a "slow" row and a "wrong answer" row for each component of a service you own.
At the intermediate level reliability becomes numbers in the design. SLOs state how good is good enough, with an error budget. Capacity is sized to survive the loss of the largest failure domain at peak. Each critical dependency has a timeout and a degraded mode agreed with product.
Types:
Uses: any service with users who notice when it is down.
Scale and failure: three zones at 60 percent become two zones at 90 percent after a zone loss, and latency climbs steeply above about 80 percent. The cost of N+1 is idle headroom every day; the trade is that headroom against minutes of a bad outage.
Try it: Intermediate
What error budget does a 99.95 percent monthly availability SLO give?
Now do it: Use the capacity simulator above with your own zone count and peak utilisation.
At the advanced level you design how much can break at once and how fast you know. Blast radius shrinks with failure domains: zones, cells, shards. Critical dependencies are found from traces and either made redundant, cached, or made optional. Alerts fire on error-budget burn rate over multiple windows, so they page early for fast burns and stay quiet for slow ones.
Types:
Uses: services others depend on, and anything whose outage is a headline.
Scale and failure: your availability is at most the product of your critical dependencies' availability in your failure domains. The cost of this level is the work of tracing, caching, and testing dependencies; the payoff is that their outages stop being yours.
Try it: Advanced
An identity service runs in one zone. Your service runs in three. What is the best design response?
Now do it: Trace one real request through your service and list every network call. Mark each one critical or non-critical.
At the master level the design includes how the system changes and how its reliability is proven. Most outages start with a change, so the design states the rollout: feature flags, canaries with automatic rollback on SLO burn, backward-compatible data changes, and a migration plan. And the design is tested by failing it on purpose: game days, zone evacuations, dependency blackholes.
Types:
Uses: every launch and every significant change of a service with an SLO.
Trade-off: progressive rollout and game days slow delivery down a little, every time. They are justified by the error budget: if most budget is being spent on changes, slower and safer change is the cheapest reliability you can buy.
Try it: Master
A canary at 5 percent shows the error rate doubling. What should happen, by design?
Now do it: Write the rollout section for your next change: stages, the metric that gates each stage, and what triggers a rollback.
The reliability section of a design fits on one page when it is written as tables. Reviewers read it before the architecture, because it is where the design admits what it does not handle.
FAILURE MODES (from a trace of GET /parcels/{id}/status)
component failure freq customer sees detect mitigation
zone lost ~1/yr nothing LB health checks N+1 per zone, replicas in all zones
read API pod crash / slow weekly nothing / slower LB, latency SLO 3 zones x 4 pods, 2 s timeout
identity svc unreachable rare nothing for 60 min call errors cached validation keys, multi-zone
status cache emptied monthly slower hit-ratio alert replicas sized for cold cache
primary DB zone loss ~1/yr writes refused ~60 s managed failover auto-promote, read-only mode
scan queue backlog seasonal stale status queue-age alert consumer autoscale, freshness SLO
SLOs (30 days) reads succeed 99.9% reads < 300 ms 99% scan visible < 60 s 99%
ALERTS page: burn 14.4x over 1 h AND 5 min ticket: burn 6x over 6 h AND 30 min
ROLLOUT flag -> 1% canary (30 min, auto-rollback on burn > 2x) -> 10% -> 50% -> 100%Built from a trace, so the identity service has a row. Every mitigation names a mechanism, not an intention.
Burn rate is the error rate divided by the error rate the SLO can sustain. A 99.9 percent SLO sustains 0.1 percent errors; at 1.44 percent errors over an hour, the burn rate is 14.4 and a 30-day budget would be gone in about two days.
# python3 burn_rate.py
SLO = 0.999
budget = 1 - SLO # 0.001 of requests may fail
windows = { # observed error ratio per window
"5m": 0.020, "1h": 0.0150, "30m": 0.004, "6h": 0.003,
}
burn = {w: e / budget for w, e in windows.items()}
page = burn["1h"] >= 14.4 and burn["5m"] >= 14.4 # fast burn, and still happening
ticket = burn["6h"] >= 6 and burn["30m"] >= 6 # slow burn, still happening
for w, b in burn.items():
print(f"{w:>3}: burn {b:5.1f}x")
days_left = 30 / burn["1h"]
print("PAGE" if page else "no page", "|", "TICKET" if ticket else "no ticket",
f"| at the 1h rate the monthly budget lasts ~{days_left:.1f} days")
Expected: burns of 20.0x, 15.0x, 4.0x, and 3.0x; then PAGE | no ticket | at the 1h rate the monthly budget lasts ~2.0 days. The short window confirms the burn is still happening, so a spike that already ended does not wake anyone.
A tracking service I reviewed after an outage had been designed carefully for zone failures: three zones, replicas everywhere, health checks that tested the database path. During a zone outage it went fully down anyway, for 47 minutes.
Two changes in tracking, one in identity. Tracking now caches the identity service's public keys and validates tokens locally for up to an hour, which turned identity from a hard dependency into a soft one. Tracking's failure table is now generated from production traces every quarter. And identity moved to three zones. The review template gained a column: dependencies, from a trace, with their failure domains.
Write the reliability section for a checkout service: it calls inventory, pricing, payments (an external provider), and a fraud check, and must take orders during a zone outage and during a payment-provider brownout.
What is the business cost of a failed checkout versus a delayed one? Which dependencies are critical, and which could fail open: is the fraud check allowed to time out to "approve" for small orders? What SLO does the business actually need? Can orders be accepted and charged later?
Say 400 checkouts a second at peak, three zones at 50 percent each, a payment provider with a p99 of 900 ms and occasional brownouts. Error budget at 99.95 percent: about 22 minutes a month.
Failure table from a trace with rows for every dependency; N+1 capacity per zone; timeouts and bulkheads per dependency; degraded modes: fraud check fails open below a value threshold, payments accepted into a queue and charged later with the customer told "processing"; SLOs and burn-rate alerts; rollout with a 1 percent canary gated on checkout success; a game day that blackholes the payment provider in staging.
Failing the fraud check open keeps revenue flowing and accepts some fraud during an outage; failing closed loses sales. Charging later keeps checkout up during a provider brownout and risks failed charges after the customer leaves. More headroom costs money every day; less makes each outage longer.