Platform Engineering · Lesson 170

Designing for Reliability and Operations: Failure Modes, SLOs, and Safe Rollout

A design that only describes the happy path is half a design. The other half answers the questions an SRE asks before go-live: what fails, how often, what the customer sees when it does, how we know, who gets woken, and how we change this thing without breaking it. Reliability is cheapest when it is designed in, as capacity headroom, failure domains, degraded modes, SLOs and alerts, and a rollout plan, and most expensive when it is discovered in the first incident. This lesson follows one tracking request while an entire availability zone is down, and shows the design decision that keeps each hop working. By the end you can write the reliability section of a design: the failure-mode table, the SLOs and their alerts, the capacity plan, and the rollout.

Difficulty: Master · Know this first: Lesson 169, Lesson 144, Lesson 43, Lesson 61

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

This bar splits one request during a zone outage. It is a worked example, not a measurement. When the design planned for the outage, the bar barely changes from a normal day; when it did not, one hop, capacity or a hidden dependency, becomes the whole bar.

Try it: is your capacity really N+1?

Move the sliders. Lose one zone and its traffic moves to the others. Watch the surviving zones' utilisation, and the API hop in the bar above.

utilisation after losing one zone = peak utilisation × zones ÷ (zones − 1) · comfortable below 80 percent; over 100 percent the survivors cannot carry the load.

error budget per 30 days = (1 − SLO) × 43,200 minutes · N+1 peak target = 80% × (zones − 1) ÷ zones

75%survivors' utilisation
53%max safe peak per zone
43 minerror budget / 30 days

N+1 holds: the survivors carry the peak

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Designing for reliability grows in four levels: first naming how each part fails, then SLOs, capacity, and degraded modes, then blast radius and critical dependencies, and finally changing a running system safely and proving the design by breaking it on purpose.

Basic — Name how every part fails

At the basic level you add a failure-mode table to the design: one row per component, with how it fails, what the customer sees, and what takes over. You find single points of failure, the boxes with no row in the "what takes over" column, and you fix them or write down why they are acceptable.

Types:

  • Crash: the process or machine stops.
  • Slowness: it answers, late, which is often worse than a crash because callers wait.
  • Wrong answers: it answers quickly and incorrectly.
  • Partial failure: one zone, one shard, one dependency, one percent of requests.

Uses: every design review and every launch checklist.

Where it breaks: a table with only crash rows. Slowness and wrong answers cause most real incidents, because nothing restarts and nothing alerts.

Try it: Basic

Which failure is usually the hardest for a design to handle?

Now do it: Add a "slow" row and a "wrong answer" row for each component of a service you own.

Intermediate — SLOs, N+1 capacity, and degraded modes

At the intermediate level reliability becomes numbers in the design. SLOs state how good is good enough, with an error budget. Capacity is sized to survive the loss of the largest failure domain at peak. Each critical dependency has a timeout and a degraded mode agreed with product.

Types:

  • SLIs: success ratio and latency percentile, measured where customers feel them.
  • SLOs and error budgets: 99.9 percent is about 43 minutes of failure a month.
  • N+1 or N+2 capacity per zone, region, or cell.
  • Degraded modes: stale with a timestamp, read-only, feature off, queue for later.

Uses: any service with users who notice when it is down.

Scale and failure: three zones at 60 percent become two zones at 90 percent after a zone loss, and latency climbs steeply above about 80 percent. The cost of N+1 is idle headroom every day; the trade is that headroom against minutes of a bad outage.

Try it: Intermediate

What error budget does a 99.95 percent monthly availability SLO give?

Now do it: Use the capacity simulator above with your own zone count and peak utilisation.

Advanced — Blast radius, dependencies, and burn-rate alerts

At the advanced level you design how much can break at once and how fast you know. Blast radius shrinks with failure domains: zones, cells, shards. Critical dependencies are found from traces and either made redundant, cached, or made optional. Alerts fire on error-budget burn rate over multiple windows, so they page early for fast burns and stay quiet for slow ones.

Types:

  • Critical versus non-critical dependencies; non-critical ones fail open.
  • Cells and shuffle sharding to limit the share of customers any failure touches.
  • Load shedding: reject the least important work first when overloaded.
  • Multi-window burn-rate alerts, for example 14.4x over one hour and 6x over six hours for a 30-day SLO.

Uses: services others depend on, and anything whose outage is a headline.

Scale and failure: your availability is at most the product of your critical dependencies' availability in your failure domains. The cost of this level is the work of tracing, caching, and testing dependencies; the payoff is that their outages stop being yours.

Try it: Advanced

An identity service runs in one zone. Your service runs in three. What is the best design response?

Now do it: Trace one real request through your service and list every network call. Mark each one critical or non-critical.

Master — Safe change, and proving the design

At the master level the design includes how the system changes and how its reliability is proven. Most outages start with a change, so the design states the rollout: feature flags, canaries with automatic rollback on SLO burn, backward-compatible data changes, and a migration plan. And the design is tested by failing it on purpose: game days, zone evacuations, dependency blackholes.

Types:

  • Progressive delivery: canary to 1 percent, then 10, then 50, gated on SLO metrics.
  • Backward compatibility: every deploy must work with the previous version's data and peers.
  • Operational readiness review: dashboards, runbooks, alerts, on-call ownership, before launch.
  • Game days and chaos experiments with a hypothesis, a blast-radius limit, and an abort switch.

Uses: every launch and every significant change of a service with an SLO.

Trade-off: progressive rollout and game days slow delivery down a little, every time. They are justified by the error budget: if most budget is being spent on changes, slower and safer change is the cheapest reliability you can buy.

Try it: Master

A canary at 5 percent shows the error rate doubling. What should happen, by design?

Now do it: Write the rollout section for your next change: stages, the metric that gates each stage, and what triggers a rollback.

Go deeper

The reliability section of a design fits on one page when it is written as tables. Reviewers read it before the architecture, because it is where the design admits what it does not handle.

FAILURE MODES (from a trace of GET /parcels/{id}/status)
component        failure          freq     customer sees           detect              mitigation
zone             lost             ~1/yr    nothing                 LB health checks    N+1 per zone, replicas in all zones
read API pod     crash / slow     weekly   nothing / slower        LB, latency SLO     3 zones x 4 pods, 2 s timeout
identity svc     unreachable      rare     nothing for 60 min      call errors         cached validation keys, multi-zone
status cache     emptied          monthly  slower                  hit-ratio alert     replicas sized for cold cache
primary DB       zone loss        ~1/yr    writes refused ~60 s    managed failover    auto-promote, read-only mode
scan queue       backlog          seasonal stale status            queue-age alert     consumer autoscale, freshness SLO

SLOs (30 days)   reads succeed 99.9%   reads < 300 ms 99%   scan visible < 60 s 99%
ALERTS           page: burn 14.4x over 1 h AND 5 min   ticket: burn 6x over 6 h AND 30 min
ROLLOUT          flag -> 1% canary (30 min, auto-rollback on burn > 2x) -> 10% -> 50% -> 100%

Built from a trace, so the identity service has a row. Every mitigation names a mechanism, not an intention.

Compute burn rates and decide whether to page

Burn rate is the error rate divided by the error rate the SLO can sustain. A 99.9 percent SLO sustains 0.1 percent errors; at 1.44 percent errors over an hour, the burn rate is 14.4 and a 30-day budget would be gone in about two days.

# python3 burn_rate.py
SLO = 0.999
budget = 1 - SLO                                   # 0.001 of requests may fail
windows = {                                        # observed error ratio per window
    "5m": 0.020, "1h": 0.0150, "30m": 0.004, "6h": 0.003,
}
burn = {w: e / budget for w, e in windows.items()}
page = burn["1h"] >= 14.4 and burn["5m"] >= 14.4   # fast burn, and still happening
ticket = burn["6h"] >= 6 and burn["30m"] >= 6      # slow burn, still happening
for w, b in burn.items():
    print(f"{w:>3}: burn {b:5.1f}x")
days_left = 30 / burn["1h"]
print("PAGE" if page else "no page", "|", "TICKET" if ticket else "no ticket",
      f"| at the 1h rate the monthly budget lasts ~{days_left:.1f} days")

Expected: burns of 20.0x, 15.0x, 4.0x, and 3.0x; then PAGE | no ticket | at the 1h rate the monthly budget lasts ~2.0 days. The short window confirms the burn is still happening, so a spike that already ended does not wake anyone.

One time it broke

A tracking service I reviewed after an outage had been designed carefully for zone failures: three zones, replicas everywhere, health checks that tested the database path. During a zone outage it went fully down anyway, for 47 minutes.

Timeline

  • 13:02Zone A loses power. The load balancer removes zone A within 20 seconds, as designed.
  • 13:03Error rate on the remaining zones goes to 100 percent. Every request fails with a token-validation timeout.
  • 13:05The burn-rate alert pages the tracking on-call. The tracking dashboards are green apart from the error rate; every component is healthy.
  • 13:20A trace shows every request waiting on the identity service, which ran only in zone A. It belonged to another team and was not in tracking's failure table.
  • 13:49Zone A recovers; so does identity, and tracking with it.

What changed after

Two changes in tracking, one in identity. Tracking now caches the identity service's public keys and validates tokens locally for up to an hour, which turned identity from a hard dependency into a soft one. Tracking's failure table is now generated from production traces every quarter. And identity moved to three zones. The review template gained a column: dependencies, from a trace, with their failure domains.

If you remember nothing else

Design exercise

Write the reliability section for a checkout service: it calls inventory, pricing, payments (an external provider), and a fraud check, and must take orders during a zone outage and during a payment-provider brownout.

Clarifying questions to ask

What is the business cost of a failed checkout versus a delayed one? Which dependencies are critical, and which could fail open: is the fraud check allowed to time out to "approve" for small orders? What SLO does the business actually need? Can orders be accepted and charged later?

Rough sizing

Say 400 checkouts a second at peak, three zones at 50 percent each, a payment provider with a p99 of 900 ms and occasional brownouts. Error budget at 99.95 percent: about 22 minutes a month.

Shape of the answer

Failure table from a trace with rows for every dependency; N+1 capacity per zone; timeouts and bulkheads per dependency; degraded modes: fraud check fails open below a value threshold, payments accepted into a queue and charged later with the customer told "processing"; SLOs and burn-rate alerts; rollout with a 1 percent canary gated on checkout success; a game day that blackholes the payment provider in staging.

Trade-offs to name

Failing the fraud check open keeps revenue flowing and accepts some fraud during an outage; failing closed loses sales. Charging later keeps checkout up during a provider brownout and risks failed charges after the customer leaves. More headroom costs money every day; less makes each outage longer.

Saved on this device, in this browser.
Next in this series · Lesson 171
Design Docs, ADRs, and Design Reviews: Writing the Decision Down

Every design in this stage ends the same way: someone has to agree to it, someone has to build it, and someone, months or years later, has to understand why it is the way it is.

Go to Lesson 171 →

Or browse the full course from the Lessons index.