One zone is a single point of failure, no matter how good the machines in it are. I have watched a whole zone go dark and take every request with it. Redundancy is the fix: a second zone standing by with the same code and the same data, ready to take over. This lesson follows one request when its zone dies mid-flight, hop by hop, the way I trace it when the pager goes off and the dashboard is all red.
Difficulty: Advanced · Know this first: Lesson 05, Lesson 12
Extra diagram · Architecture map
One zone serving, one zone standing by. I start with one zone and build up to the full standby picture.
Extra diagram · Second trip
A zone dies in the middle of a request. I follow that request from the first missed probe to the answer it finally gets.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a normal request the work is cheap, and in a failover the detection, the shift, and the warm-up eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.
No standby. The request fails outright. There is nothing to shift to.
The numbers above are one fixed example. Move the controls below and watch the same failover change. The budget bar, the timings on the diagram, and the status lines all update live. The simulator starts from the clean failover scenario. Switching to a different scenario resets these controls to their defaults, so the two never fight each other.
How many standby zones sit behind the primary. Zones do not change the failover time for one zone failure. They change how many zone failures the system survives. Zero means no failover at all.
Three missed probes in a row call the zone dead, so detection is three times the interval. At 1 second, detection is 3,000 ms. At 10 seconds, it is 30,000 ms.
The shift is 500 ms plus the TTL. At a TTL of 0 the traffic manager flips the route itself. At 300 seconds, clients hold the old answer and the shift is 300,500 ms.
A warm standby that serves real traffic needs about 1.5 seconds. A cold standby that boots from nothing needs minutes. At 120 seconds, warm-up is 120,000 ms.
Detection 3,000 ms + shift 500 ms + warm-up 1,500 ms. Total trip: ~5,130 ms.
Failover lands in seconds. Users notice a pause, not an outage.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times. First the spare copy, then the fast switchover, then the ways the switchover itself fails, then the second region.
Definition: Redundancy means a second copy of everything that can die. The zone, the data, and the route to it. I run the same code in a second zone with the same data, standing by. When the first zone goes dark, the second one takes the traffic. Failover is the move itself. Detection, shift, warm-up. Three pieces, and each one is a setting I chose on a calm day.
Types:
Uses: I use it for the day a whole zone goes dark. I have watched that happen. The request in flight dies with the zone, and the next request has to land somewhere else. I also use it for planned work. A zone drained for maintenance is a failover with a calendar invite instead of a pager.
Dimensions: Active-passive costs double the machines for the same traffic, and the standby sits near idle. That is the deal. I pay for capacity I hope never to use, the same way I pay for insurance. Failover time is detection plus shift plus warm-up. A cold standby can take three minutes to answer the first request. The trade is money for uptime. The standby earns its keep the one day the primary dies.
Definition: A fast failover is three numbers I control. How often the health check probes, how many misses call a zone dead, and how long a DNS answer lives. I probe every second and call the zone dead after three misses, so detection takes about three seconds. I shift traffic at the traffic manager, not through DNS, because a DNS TTL of 300 seconds is a five-minute failover written down in advance.
Types:
Uses: I use this when I am setting the failover numbers for a real service and when I am drilling the failover on a schedule. I read the three numbers on a bad day before I trust any dashboard. Dashboards show the failover happened. The config shows how long it was always going to take.
Dimensions: As load grows, the standby has to absorb the full load of the primary in seconds. If the standby is sized at half the primary, it falls over the moment traffic arrives. I size each zone to carry the full load alone. The failure I watch for is flapping. A health check that is too sensitive flips traffic back and forth, and users get a different answer on every retry. The cost is drill time and 5 percent of traffic. The trade is detection speed against stability. A hold time after a flip lets the system settle before it flips again.
Definition: The failover can work perfectly and the system can still go down. The standby takes over with data that is 40 seconds old, and users see their own recent writes vanish. Or both zones think they are the primary and both take writes, and the writes conflict when the zones meet again. Or the standby has a different config, a different secret, a different version, because the last config change went to the primary and never went to the standby. The failover worked. The system still went down.
Types:
Uses: I use this in the first five minutes of an incident, when I check the lag reading before I promote and write down the gap. I would rather tell the team the number of lost writes in the first five minutes than discover it from users the next day. I use it in reviews, when I ask when the failover was last drilled. A drill nobody runs catches nothing.
Dimensions: At ten times the traffic, one standby zone stops being enough for the failover itself, and the traffic manager becomes the thing I watch, because one global entry point is itself a single point of failure. The cost is fencing, lag checks, and a monthly drill in a staging zone that looks like production. The trade is promotion speed against data safety. Promoting fast with stale data is a data loss plan with extra steps. Promotion without fencing is not failover.
Definition: At this level the failure is not one zone. It is a whole region going dark. When the us-east-1 region had its big outage in December 2021, the services that survived were the ones whose standby was warm and tested, and the ones that went down hardest were the ones whose redundancy stopped at the region boundary. A second region means copies of the data in another region, async replication across regions, and a recovery time measured in minutes instead of seconds.
Types:
Uses: I use this when one region is a single point of failure for the business, not just a zone. I use it when the recovery target is minutes and the budget allows a second region standing by. And I use it when I have to say plainly that a user reading from the far region is reading the past by design, and the design has to own that.
Dimensions: Cross-region replication lag is measured in seconds or minutes, not milliseconds, and that lag is the data loss on a region failover. The cost is a whole second region, plus the team that keeps it warm. The failure mode is a region failover that works and still loses the last minute of writes, because async replication had not caught up. The trade is consistency against distance. I keep sync replication inside one region and async across regions, and I write the recovery time down before the bad day, not during it.
The lesson above is the shape of a clean failover. Production is messier. Here is what I actually argue about when I set this up for real.
Active-passive is the simple version. One zone serves traffic. The other waits. It costs double the machines for the same traffic, and the standby sits near idle. That is the deal. I pay for capacity I hope never to use, the same way I pay for insurance. The upside is the standby is simple to reason about. There is one writer, one source of truth, and no fights over who owns a write.
Active-active is the expensive version of the same idea. Both zones serve traffic all the time. Utilization looks good on the dashboard and nobody asks why a zone sits idle. The cost moves somewhere else. Now two zones take writes, and writes in two places have to agree. That means conflict rules, or a single writer with cross zone latency, or accepting that two users can book the same seat at the same second. I only go active-active when the traffic justifies it and the data can take it. For most systems, active-passive with a warm standby is the right call, and the money saved on complexity pays for the idle machines.
The middle path is the one I actually use most. The standby serves a small slice of real traffic, about 5 percent. It stays warm, its caches stay hot, and I see its errors in the dashboard every day instead of discovering them during an outage. It costs a little routing work. It saves me from the cold standby surprise.
Split brain is the one I plan for and hope never to see. Two zones both think they are the primary, and both take writes. When they meet again, the writes conflict and somebody loses. The fix is fencing. The old primary must be stopped from taking writes before the new one starts. Promotion without fencing is not failover. It is a data loss plan with extra steps.
Replication lag at the moment of failover is quieter and more common. The standby takes over with data that is 40 seconds old. Users see their own recent writes vanish. I check the lag reading before I promote, and I write down the gap. That gap is the list of writes I lost. I would rather tell the team that number in the first five minutes than discover it from users the next day.
A failover that never got tested is the most common failure of all. The runbook says it takes 30 seconds. The runbook was written two years ago. The standby has a different config now, a different secret, a different version. I test failover on a schedule, not on hope. A monthly drill in a staging zone that looks like production catches the drift. A drill nobody runs catches nothing.
Flapping is the failure that looks like success. The health check is too sensitive, so the traffic manager flips to the standby, then back to the primary, then to the standby again. Users get a different answer on every retry. I set the miss threshold high enough that a slow probe does not look like a dead zone, and I add a hold time after a flip so the system settles before it flips again.
At 10x the traffic, one standby zone stops being enough for the failover itself. The standby has to absorb the full load of the primary in seconds. If the standby is sized at half the primary, it falls over the moment traffic arrives. I size each zone to carry the full load alone, not half of it. That is the same rule as replicas in Lesson 12. Every copy must carry the whole load by itself.
At 10x the zones, the traffic manager becomes the thing I watch. One global entry point that routes to many zones is itself a single point of failure, so I run it in more than one place too, with health checks on the checkers. Redundancy has to cover the redundancy.
At 100x, failover stops being a zone problem and becomes a region problem. A whole region can go dark, not just a zone. That means copies of the data in a second region, async replication across regions from Lesson 12, and a recovery time measured in minutes instead of seconds. I keep sync replication inside one region and async across regions, because the speed of light does not negotiate. A user reading from the far region is reading the past by design, and the design has to own that.
A short health check and failover policy, in the shape I actually keep in a runbook. The names are generic. The point is the three numbers that decide the failover time: how often to probe, how many misses call it dead, and how long a DNS answer lives.
# Health check: probe every 1s, call the zone dead after 3 misses. # Detection time is 3s. Shorten the interval and detection gets # faster. Shorten it too far and a slow probe looks like a dead zone. health_check: interval_seconds: 1 missed_probes_to_fail: 3 # Failover: shift traffic at the traffic manager, not through DNS. # A DNS TTL of 300s is a 5 minute failover written down in advance. failover: mode: traffic_manager dns_ttl_seconds: 0 # Standby: keep it warm with 5 percent of real traffic. # A standby that never serves a request is a standby you never tested. standby: live_traffic_percent: 5
I read those three numbers on a bad day before I trust any dashboard. Dashboards show the failover happened. The config shows how long it was always going to take.
A whole zone went dark on a weekday afternoon. The traffic manager noticed, the shift happened, and the standby took over. That part worked. Then every request on the standby started failing. Not slow. Failing. The standby pods were running and healthy. They were calling an internal address that only existed in the dead zone. A config change months earlier had pointed a downstream service at the primary zone's private address. The change went to the primary. It never went to the standby. The standby had never served a real request, so nobody noticed. The failover worked perfectly and the system still went down.
Three things changed after, and I still do them now. First, the standby serves 5 percent of real traffic every day, so a wrong config shows up on a quiet Tuesday instead of during an outage. Second, config deploys go to every zone in one pipeline. A change that reaches only one zone is a failed deploy, and the pipeline says so. Third, failover gets drilled once a month in a staging zone that looks like production. The drill is not a demo. It is a test, and it fails until it passes.
Design failover for a checkout API across two zones. The API takes 2,000 requests per second at peak. Recovery time objective is 60 seconds: the API must be answering again within a minute of a zone dying. Recovery point objective is near zero: a completed checkout must never vanish. The checkout writes to a database, and the database already replicates to the standby zone.
Clarifying questions to ask first
Worked approach
Trade-offs to name