Platform Engineering · Lesson 41

SRE Skills

A pager goes off at 2 AM. That is the moment every SRE skill gets tested at once. Triage, mitigation, communication, the fix, the automation, the math, the planning, the review. None of them is hard on its own. The hard part is doing them in the right order, under pressure, when you are tired and the checkout is down. This lesson follows one alert from the moment it fires to the moment the toil it created is automated away. You will see the week where it works, the week where the pager never stops, and the week where there is no time left to automate anything. The difference between those weeks is not talent. It is which skills get practiced and which ones get skipped.

Difficulty: Advanced · Know this first: Lesson 25, Lesson 35
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in minutes. The bar below is a worked example for one alert's week, not a measurement from a live on-call rotation, but the shape is honest: in a normal week the biggest slice is the automation hop, because that is where the week pays for itself. In a storm week the biggest slice is the fix, and the automation slice disappears. In a toil week the whole bar shrinks, and that is the trap. Less time this week, the same time again next week, forever. Step through the lesson or switch the scenario and watch the bar change.

Turn the knobs and watch the budget and the toil move

The budget above is one fixed week. This box is yours. Push the SLO target to 99.99 percent and watch the error budget shrink to minutes. Push the toil slider past 20 hours and watch the verdict flip. Then ask yourself the question I ask every Monday: how much of my week is toil, and what would I automate first if I had the time back?

How reliable the service promises to be. A higher target means a smaller error budget.

How much traffic the error budget covers. It drives how many failed requests are allowed.

How much of the week is manual, repetitive work. It drives the verdict below.

Error budget / month43.2 minat a 99.9% SLO
Allowed failed requests / month100,000out of 100M requests
Toil as % of a 40-hour week30%12 of 40 hours

Rate yourself honestly

Five skills, one to five each. Be honest. A three means you can do it with a runbook open. A five means you can do it at 2 AM with the pager going off.

Can you read a dashboard and know what is wrong?

Can you triage, mitigate, and communicate under pressure?

Can you turn manual work into a script that runs itself?

Can you say when the service runs out of room?

Can you work out an error budget and a burn rate?

Total score15 / 25five skills, five points each
Weakest skillEven at 3all five are equal right now

Assumptions, stated plainly: a 30-day month has 43,200 minutes, and a working week is 40 hours. Both are round teaching numbers. The error budget is (100 − SLO%) of the minutes in the month, and the allowed failures are the same fraction of the requests in the month.

The on-call system, from scratch: one alert, one human

Extra diagram. I read the picture first: the boxes are the system, the moving dot is an alert, and each step adds one piece until the full picture is there. No text needed to follow it.

99.9% over 30 days = 43.2 minutes of error budgetToil capped at half the weekBudget freeze backed by leadership
1234567891011

Human stepsPolicy decidesMaster guardrailsMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The lesson above follows one alert through nine hops. This section covers the same job four times, from the first page to the level where you run reliability for a whole team and still sleep.

Basic — What the job is

Definition: An SRE keeps the service up and makes the next outage cheaper. I triage, I mitigate, I communicate, I fix, and I automate. None of those is a superpower. The job is doing them in the right order, under pressure, when I am tired and the checkout is down.

Types:

  • Triage — decide if the alert is real and how bad it is.
  • Mitigation — stop the bleeding. Roll back, shed traffic, fail over.
  • Communication — say what is broken, who is affected, and when the next update comes.
  • The fix — remove the cause so the deploy can go out again.
  • Automation — turn the manual work into something that runs itself.

Uses: I use this level every time the pager fires. It is the shape of a normal on-call week. One alert, worked cleanly, from page to review. If I skip a step, the next page is the same alert wearing a different time stamp.

Dimensions:

  • Load: At one alert a week, this works by memory and a runbook. At ten alerts a night, memory fails and only the runbook and the alert routing hold.
  • Breaks: It breaks when triage is skipped. I fix the wrong thing, or I roll back a good deploy while the real cause sits untouched. I notice it when the same alert fires again next week.
  • Cost: The cost is my night and my morning. A clean alert costs about four hours. A messy one costs the night and the next day, because a tired engineer writes the next bug.
  • Trade-off: I give up speed to get order. Mitigating before fixing feels slow. It is slow. It is also the only way the users stop hurting while I think.

Intermediate — SLOs and error budgets

Definition: An SLO is a written promise: this much of the service works, this fast, over this long. The error budget is what is left when the promise is kept. At 99.9 percent over 30 days, the budget is 43.2 minutes of badness a month. That number decides what happens next.

Types:

  • Availability SLO — the share of requests that succeed.
  • Latency SLO — the share that answer fast enough.
  • Burn-rate alerts — fast burn pages, slow burn opens a ticket.
  • The freeze rule — budget gone means feature deploys stop.

Uses: I use this level the morning after an incident, and every Monday. The budget says whether the team ships or fixes. I say it with the numbers, not with a feeling. Forty minutes burned by day ten is a fact the product team can act on.

Dimensions:

  • Load: The budget math does not change with traffic. The burn rate does. The same incident at ten times the traffic hurts ten times as many users in the same minutes.
  • Breaks: It breaks when the SLO measures the wrong thing. If the SLO counts a request as good when the user saw an error page, the budget is a lie. I notice it when users complain and the dashboard is green.
  • Cost: The cost is discipline. Measuring the SLO every week, reviewing it every quarter, and holding the freeze when the budget is gone. A freeze nobody enforces is a poster.
  • Trade-off: I give up shipping speed to get reliability. A tighter SLO looks better on a slide and burns the team out. I set 99.9, not 99.99, because one incident should not end the month.

Advanced — Toil and a rotation that lasts

Definition: Toil is work that is manual, repetitive, automatable, and has no lasting value. Restarting the same stuck job by hand is toil. Writing the script that restarts it is engineering. I cap toil at half the week and I count it, the same way I count the error budget.

Types:

  • The toil budget — toil over half the week means feature work stops and automation starts.
  • The rotation — one week on, three weeks off, primary and secondary, written handoff.
  • The protected day — one day a week with no meetings, for automation only.
  • The review — every incident gets owners and dates, checked a week later.

Uses: I use this level when the pager never stops. The week the same alert fires six times is not a bad week. It is a system telling me the automation never happened. I fix the system, not the week.

Dimensions:

  • Load: Toil scales with traffic. Manual work that was a nuisance at one size becomes a full-time job at ten times the size. Automation stops being nice to have. It becomes survival.
  • Breaks: It breaks when leadership overrides the toil budget every sprint. I notice it when the protected day fills with meetings and the same manual work comes back next week.
  • Cost: The cost is a day a week of engineering time, and the courage to stop shipping while the team automates. It pays back the next time the alert fires and nobody has to wake up.
  • Trade-off: I give up feature speed to get a team that lasts. A rotation where one person gets every page is not a rotation. It is one person burning out while the others watch.

Master — Reliability as a policy

Definition: At this level reliability is not a habit. It is a policy the organisation enforces. The error-budget freeze is backed by leadership. The toil budget is a line in the roadmap. On-call runs across time zones so nobody is paged at 3 AM every night. I stop persuading and start measuring.

Types:

  • The policy file — SLO, budget, burn alerts, and freeze rule in one place, argued with numbers.
  • Cross-time-zone on-call — follow-the-sun, so the night is shared.
  • The review as the most important meeting — one unlearned lesson repeats a hundred times as fast at scale.
  • Quarterly SLO review — too easy means over-investing, too hard means the service needs real work.

Uses: I use this level when one team's reliability becomes the company's reliability. A hundred times the traffic means the error-budget policy is enforced by leadership, not by goodwill. The review changes things, or the review is a story.

Dimensions:

  • Load: At a hundred times the traffic, SRE stops being a rotation and becomes a team. The pager fires more often, the handoff becomes a real document, and the toil budget is staffed like feature work.
  • Breaks: It breaks when the policy is written but nobody enforces it. I notice it when the budget is gone and the deploy goes out anyway, and nobody says anything.
  • Cost: The cost is headcount and honesty. A real rotation needs enough people. A real policy needs a leader who will hold a freeze in front of a deadline.
  • Trade-off: I give up the freedom to improvise to get a system that survives me. My judgment becomes a document, a dashboard, and a rule. That is the point. The service should not depend on me being awake.

Go deeper

The lesson above is one alert through nine hops. Production is the rest of it. Here is what I actually spend my time on once the pager, the dashboards, and the runbooks are in place.

SLO and error-budget policy

I write the SLO down before I write the alert. The SLO says what good looks like: 99.9 percent of checkout requests succeed in under 500 milliseconds, measured over 30 days. The error budget is what is left: 43.2 minutes a month of badness. The policy says what happens when the budget is gone: feature deploys stop, and reliability work starts. I alert on the burn rate, not just the total. A fast burn, two percent of the budget in one hour, pages me. A slow burn, the budget draining over three weeks, goes to the weekly review. Both matter, and they need different responses. I also review the SLO every quarter. If we never burn the budget, the SLO is too easy and we are over-investing in reliability. If we burn it every month, the SLO is too hard or the service needs real work. Either way, the SLO is a decision, not a decoration.

Toil budget

I cap toil at half the week, and I measure it. Toil is work that is manual, repetitive, automatable, tactical, and has no lasting value. Restarting a stuck job by hand is toil. Writing the script that restarts it is engineering. I count toil hours every week, in the same review where I count the error budget. If toil goes over half the week, feature work stops and automation work starts, the same way deploys stop when the error budget is gone. The rule only works if leadership backs it. A toil budget that a manager overrides every sprint is not a budget. It is a suggestion, and suggestions do not survive a deadline.

On-call rotation

I run a rotation with a primary and a secondary, one week on and at least three weeks off. Nobody does two weeks in a row. The handoff is a written note, not a chat message: what is flaky, what is being watched, what changed this week. Pages go to the primary first, and the secondary only if the primary does not answer in five minutes. I pay attention to who gets paged most, because the rotation is only fair if the load is shared. A rotation where one person gets every page is not a rotation. It is one person burning out while the others watch. I also protect the day after a bad night. A person who was paged three times does not do a full day of feature work the next morning. Tired engineers write the bugs that page the next person.

What changes at 10x

At ten times the traffic, the error budget math does not change, but the burn rate does. The same incident burns the same minutes, and the minutes are worth more, because ten times as many users feel them. Toil scales with traffic, so the manual work that was a nuisance becomes a full-time job, and automation stops being nice to have. It becomes survival. The rotation needs more people, because the pager fires more often, and one person cannot hold it. The SLOs need to be re-negotiated, because a target that was easy at one size can be impossible at ten times the size. I redo the SLO math before the growth arrives, not after the budget is gone.

What changes at 100x

At a hundred times the traffic, SRE stops being a rotation and becomes a team. The error-budget policy becomes an organisation policy, not a team habit, and it gets enforced by leadership, not by goodwill. On-call runs across time zones, so nobody is paged at 3 AM every night, and the handoff becomes a real process with a real document. The toil budget is a line in the roadmap, not a hope. Automation work gets planned and staffed like feature work, because at this size the toil would eat the whole team otherwise. And the review process from Step 9 becomes the most important meeting in the week, because at a hundred times the traffic, one unlearned lesson repeats a hundred times as fast.

SLO definition I actually use

This is a generic SLO definition, not tied to any one vendor:

# SLO: checkout availability. The promise, the budget,
# and what happens when the budget is gone.
service: checkout
slo:
  name: checkout-availability
  target: 99.9  # percent of good requests, over 30 days
  # 99.9% leaves 0.1% bad. 0.1% of 43,200 minutes
  # in a 30-day month is 43.2 minutes. That is the budget.
  window_days: 30
  error_budget_minutes: 43.2
  good_request:
    # A request counts as good when it succeeds
    # and answers in under 500 ms.
    status: 2xx
    latency_ms: 500
alerts:
  # Fast burn: 2% of the monthly budget in one hour. Page.
  - name: fast-burn
    burn_rate: 2% of budget in 1h
    action: page
  # Slow burn: the budget draining over three weeks. Ticket.
  - name: slow-burn
    burn_rate: 10% of budget in 3 days
    action: ticket
policy:
  # Budget gone: feature deploys stop until it recovers.
  budget_exhausted: freeze feature deploys

The definition is the whole policy in one file. The target, the budget, the burn alerts, and the freeze rule. A team that can point at this file can argue about the numbers instead of arguing about whether the deploy should go out. That argument, with numbers, is the job.

One time it broke

This one stayed with me because nobody did anything wrong, and the team still nearly broke. The pager was not the problem. The toil was.

What broke

A team of four SREs was paged every night for months. The same alerts, the same manual restarts, the same checks run by hand. Each incident was small. Nothing ever went down for long. But the work never shrank, because there was never time to automate it. Every week the manual work filled the week, so the automation never happened, so the manual work came back the next week. By the third month the error budget was burned by day 12, feature deploys were frozen, and two of the four were talking about leaving. The system was up. The team was down. That is the incident I remember, because the dashboards were green the whole time.

Timeline

  • Week 1A batch job starts failing every night at 1 AM. Someone restarts it by hand. It takes twenty minutes. Nobody writes it down, because it is small.
  • Week 3The same job fails three nights in a row. Three different people restart it. Nobody notices it is the same job, because there is no shared log of manual work.
  • Week 5The pager fires eleven times in one week. Four of the pages are the batch job. The team starts calling it the nightly page, like it is weather.
  • Week 8The error budget for the month is burned by day 12. Feature deploys freeze. The product team asks why, and the answer is a list of small incidents nobody fixed properly.
  • Week 10One engineer is paged four nights in a row and takes a week off. The remaining three split the pager, and the manual work gets worse, because there are fewer hands for the same toil.
  • Week 12A review finally happens. The toil log shows 22 hours a week of manual restarts and checks across the team. More than half the week. The team agrees on three changes, and they stick to them this time.

What changed after

Three things, and I still do all three. First, a paging policy. Only actionable alerts page. The batch job failure became a ticket with an automated restart, and it stopped paging anyone. The pager went from eleven pages a week to three. Second, a toil budget, counted every week. Toil over half the week means feature work stops and automation work starts, and the manager backs it. Third, one protected day a week for automation, no meetings, no pages unless the service is down. The batch job got its automated restart and its fix in the first protected day. The error budget stopped burning by day 12. The team did not get bigger. The work got smaller, and that was the fix.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I take a real service on call. Take a minute with it before you read my approach.

The prompt

You run a checkout service at 500 requests per second at peak. It has no SLOs and no on-call rotation. The team is four engineers, and the pager goes to whoever is awake. Design the on-call rotation and the SLO set for this service. What do you measure, what pages you, who carries the pager, and what happens when the budget is gone?

Clarifying questions to ask first

I would ask five things before I sized anything. What is the current failure rate, and what does a bad hour look like in numbers? What is the traffic pattern: is 500 requests a second the peak, the average, or the whole day? How many pages does the team get in a week today, and how many of them are actionable? What is the recovery time when checkout goes down: minutes, or hours? And what does the business lose per minute of checkout downtime: is this a store where every minute is money, or a service where an hour is an annoyance?

Rough sizing, worked

500 requests a second is 43.2 million requests a day, and about 1.3 billion a month. At 99.9 percent availability, the error budget is 43.2 minutes a month, and the allowed failures are about 1.3 million requests a month. That sounds like a lot until one bad deploy burns 20 minutes in a night. I set two SLOs. Availability: 99.9 percent of requests succeed. Latency: 99 percent of successful requests answer in under 500 milliseconds. The rotation is four people, one week on and three weeks off, with a primary and a secondary. The primary carries the pager. The secondary answers only if the primary does not answer in five minutes. The pager fires only for two things: the error budget burning fast, and checkout availability dropping below the SLO. Everything else is a ticket. The handoff is a written note every Monday: what is flaky, what is being watched, what changed. And the day after a bad night is protected. No feature work, no meetings, just recovery and the review.

Trade-offs to name

I set the SLO at 99.9, not 99.99, because 99.99 leaves 4.32 minutes a month, and one incident ends the month. A tighter SLO looks better on a slide and burns the team out in practice. I page on two things only, and I accept that some real problems will arrive as tickets instead of pages, because a pager that fires for everything is a pager nobody trusts. I keep the rotation at one week in four, and I accept that it is slower to staff than giving the pager to whoever is awake, because a fair rotation is the only one that lasts. And I spend the first month measuring before I tighten anything. An SLO set from guesses is a guess, and the numbers from the first month are the real answer.

Next in this series
Lesson 42 — Root Cause Analysis