A pager goes off at 2 AM. That is the moment every SRE skill gets tested at once. Triage, mitigation, communication, the fix, the automation, the math, the planning, the review. None of them is hard on its own. The hard part is doing them in the right order, under pressure, when you are tired and the checkout is down. This lesson follows one alert from the moment it fires to the moment the toil it created is automated away. You will see the week where it works, the week where the pager never stops, and the week where there is no time left to automate anything. The difference between those weeks is not talent. It is which skills get practiced and which ones get skipped.
Every hop charges rent in minutes. The bar below is a worked example for one alert's week, not a measurement from a live on-call rotation, but the shape is honest: in a normal week the biggest slice is the automation hop, because that is where the week pays for itself. In a storm week the biggest slice is the fix, and the automation slice disappears. In a toil week the whole bar shrinks, and that is the trap. Less time this week, the same time again next week, forever. Step through the lesson or switch the scenario and watch the bar change.
The budget above is one fixed week. This box is yours. Push the SLO target to 99.99 percent and watch the error budget shrink to minutes. Push the toil slider past 20 hours and watch the verdict flip. Then ask yourself the question I ask every Monday: how much of my week is toil, and what would I automate first if I had the time back?
How reliable the service promises to be. A higher target means a smaller error budget.
How much traffic the error budget covers. It drives how many failed requests are allowed.
How much of the week is manual, repetitive work. It drives the verdict below.
Five skills, one to five each. Be honest. A three means you can do it with a runbook open. A five means you can do it at 2 AM with the pager going off.
Can you read a dashboard and know what is wrong?
Can you triage, mitigate, and communicate under pressure?
Can you turn manual work into a script that runs itself?
Can you say when the service runs out of room?
Can you work out an error budget and a burn rate?
Assumptions, stated plainly: a 30-day month has 43,200 minutes, and a working week is 40 hours. Both are round teaching numbers. The error budget is (100 − SLO%) of the minutes in the month, and the allowed failures are the same fraction of the requests in the month.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is an alert, and each step adds one piece until the full picture is there. No text needed to follow it.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above follows one alert through nine hops. This section covers the same job four times, from the first page to the level where you run reliability for a whole team and still sleep.
Definition: An SRE keeps the service up and makes the next outage cheaper. I triage, I mitigate, I communicate, I fix, and I automate. None of those is a superpower. The job is doing them in the right order, under pressure, when I am tired and the checkout is down.
Types:
Uses: I use this level every time the pager fires. It is the shape of a normal on-call week. One alert, worked cleanly, from page to review. If I skip a step, the next page is the same alert wearing a different time stamp.
Dimensions:
Definition: An SLO is a written promise: this much of the service works, this fast, over this long. The error budget is what is left when the promise is kept. At 99.9 percent over 30 days, the budget is 43.2 minutes of badness a month. That number decides what happens next.
Types:
Uses: I use this level the morning after an incident, and every Monday. The budget says whether the team ships or fixes. I say it with the numbers, not with a feeling. Forty minutes burned by day ten is a fact the product team can act on.
Dimensions:
Definition: Toil is work that is manual, repetitive, automatable, and has no lasting value. Restarting the same stuck job by hand is toil. Writing the script that restarts it is engineering. I cap toil at half the week and I count it, the same way I count the error budget.
Types:
Uses: I use this level when the pager never stops. The week the same alert fires six times is not a bad week. It is a system telling me the automation never happened. I fix the system, not the week.
Dimensions:
Definition: At this level reliability is not a habit. It is a policy the organisation enforces. The error-budget freeze is backed by leadership. The toil budget is a line in the roadmap. On-call runs across time zones so nobody is paged at 3 AM every night. I stop persuading and start measuring.
Types:
Uses: I use this level when one team's reliability becomes the company's reliability. A hundred times the traffic means the error-budget policy is enforced by leadership, not by goodwill. The review changes things, or the review is a story.
Dimensions:
The lesson above is one alert through nine hops. Production is the rest of it. Here is what I actually spend my time on once the pager, the dashboards, and the runbooks are in place.
I write the SLO down before I write the alert. The SLO says what good looks like: 99.9 percent of checkout requests succeed in under 500 milliseconds, measured over 30 days. The error budget is what is left: 43.2 minutes a month of badness. The policy says what happens when the budget is gone: feature deploys stop, and reliability work starts. I alert on the burn rate, not just the total. A fast burn, two percent of the budget in one hour, pages me. A slow burn, the budget draining over three weeks, goes to the weekly review. Both matter, and they need different responses. I also review the SLO every quarter. If we never burn the budget, the SLO is too easy and we are over-investing in reliability. If we burn it every month, the SLO is too hard or the service needs real work. Either way, the SLO is a decision, not a decoration.
I cap toil at half the week, and I measure it. Toil is work that is manual, repetitive, automatable, tactical, and has no lasting value. Restarting a stuck job by hand is toil. Writing the script that restarts it is engineering. I count toil hours every week, in the same review where I count the error budget. If toil goes over half the week, feature work stops and automation work starts, the same way deploys stop when the error budget is gone. The rule only works if leadership backs it. A toil budget that a manager overrides every sprint is not a budget. It is a suggestion, and suggestions do not survive a deadline.
I run a rotation with a primary and a secondary, one week on and at least three weeks off. Nobody does two weeks in a row. The handoff is a written note, not a chat message: what is flaky, what is being watched, what changed this week. Pages go to the primary first, and the secondary only if the primary does not answer in five minutes. I pay attention to who gets paged most, because the rotation is only fair if the load is shared. A rotation where one person gets every page is not a rotation. It is one person burning out while the others watch. I also protect the day after a bad night. A person who was paged three times does not do a full day of feature work the next morning. Tired engineers write the bugs that page the next person.
At ten times the traffic, the error budget math does not change, but the burn rate does. The same incident burns the same minutes, and the minutes are worth more, because ten times as many users feel them. Toil scales with traffic, so the manual work that was a nuisance becomes a full-time job, and automation stops being nice to have. It becomes survival. The rotation needs more people, because the pager fires more often, and one person cannot hold it. The SLOs need to be re-negotiated, because a target that was easy at one size can be impossible at ten times the size. I redo the SLO math before the growth arrives, not after the budget is gone.
At a hundred times the traffic, SRE stops being a rotation and becomes a team. The error-budget policy becomes an organisation policy, not a team habit, and it gets enforced by leadership, not by goodwill. On-call runs across time zones, so nobody is paged at 3 AM every night, and the handoff becomes a real process with a real document. The toil budget is a line in the roadmap, not a hope. Automation work gets planned and staffed like feature work, because at this size the toil would eat the whole team otherwise. And the review process from Step 9 becomes the most important meeting in the week, because at a hundred times the traffic, one unlearned lesson repeats a hundred times as fast.
This is a generic SLO definition, not tied to any one vendor:
# SLO: checkout availability. The promise, the budget,
# and what happens when the budget is gone.
service: checkout
slo:
name: checkout-availability
target: 99.9 # percent of good requests, over 30 days
# 99.9% leaves 0.1% bad. 0.1% of 43,200 minutes
# in a 30-day month is 43.2 minutes. That is the budget.
window_days: 30
error_budget_minutes: 43.2
good_request:
# A request counts as good when it succeeds
# and answers in under 500 ms.
status: 2xx
latency_ms: 500
alerts:
# Fast burn: 2% of the monthly budget in one hour. Page.
- name: fast-burn
burn_rate: 2% of budget in 1h
action: page
# Slow burn: the budget draining over three weeks. Ticket.
- name: slow-burn
burn_rate: 10% of budget in 3 days
action: ticket
policy:
# Budget gone: feature deploys stop until it recovers.
budget_exhausted: freeze feature deploys
The definition is the whole policy in one file. The target, the budget, the burn alerts, and the freeze rule. A team that can point at this file can argue about the numbers instead of arguing about whether the deploy should go out. That argument, with numbers, is the job.
This one stayed with me because nobody did anything wrong, and the team still nearly broke. The pager was not the problem. The toil was.
A team of four SREs was paged every night for months. The same alerts, the same manual restarts, the same checks run by hand. Each incident was small. Nothing ever went down for long. But the work never shrank, because there was never time to automate it. Every week the manual work filled the week, so the automation never happened, so the manual work came back the next week. By the third month the error budget was burned by day 12, feature deploys were frozen, and two of the four were talking about leaving. The system was up. The team was down. That is the incident I remember, because the dashboards were green the whole time.
Three things, and I still do all three. First, a paging policy. Only actionable alerts page. The batch job failure became a ticket with an automated restart, and it stopped paging anyone. The pager went from eleven pages a week to three. Second, a toil budget, counted every week. Toil over half the week means feature work stops and automation work starts, and the manager backs it. Third, one protected day a week for automation, no meetings, no pages unless the service is down. The batch job got its automated restart and its fix in the first protected day. The error budget stopped burning by day 12. The team did not get bigger. The work got smaller, and that was the fix.
This is the kind of question I would ask in an interview, and the kind I ask myself before I take a real service on call. Take a minute with it before you read my approach.
You run a checkout service at 500 requests per second at peak. It has no SLOs and no on-call rotation. The team is four engineers, and the pager goes to whoever is awake. Design the on-call rotation and the SLO set for this service. What do you measure, what pages you, who carries the pager, and what happens when the budget is gone?
I would ask five things before I sized anything. What is the current failure rate, and what does a bad hour look like in numbers? What is the traffic pattern: is 500 requests a second the peak, the average, or the whole day? How many pages does the team get in a week today, and how many of them are actionable? What is the recovery time when checkout goes down: minutes, or hours? And what does the business lose per minute of checkout downtime: is this a store where every minute is money, or a service where an hour is an annoyance?
500 requests a second is 43.2 million requests a day, and about 1.3 billion a month. At 99.9 percent availability, the error budget is 43.2 minutes a month, and the allowed failures are about 1.3 million requests a month. That sounds like a lot until one bad deploy burns 20 minutes in a night. I set two SLOs. Availability: 99.9 percent of requests succeed. Latency: 99 percent of successful requests answer in under 500 milliseconds. The rotation is four people, one week on and three weeks off, with a primary and a secondary. The primary carries the pager. The secondary answers only if the primary does not answer in five minutes. The pager fires only for two things: the error budget burning fast, and checkout availability dropping below the SLO. Everything else is a ticket. The handoff is a written note every Monday: what is flaky, what is being watched, what changed. And the day after a bad night is protected. No feature work, no meetings, just recovery and the review.
I set the SLO at 99.9, not 99.99, because 99.99 leaves 4.32 minutes a month, and one incident ends the month. A tighter SLO looks better on a slide and burns the team out in practice. I page on two things only, and I accept that some real problems will arrive as tickets instead of pages, because a pager that fires for everything is a pager nobody trusts. I keep the rotation at one week in four, and I accept that it is slower to staff than giving the pager to whoever is awake, because a fair rotation is the only one that lasts. And I spend the first month measuring before I tighten anything. An SLO set from guesses is a guess, and the numbers from the first month are the real answer.