Platform Engineering · Lesson 22

When One Fails, Another Steps In

One zone is a single point of failure, no matter how good the machines in it are. I have watched a whole zone go dark and take every request with it. Redundancy is the fix: a second zone standing by with the same code and the same data, ready to take over. This lesson follows one request when its zone dies mid-flight, hop by hop, the way I trace it when the pager goes off and the dashboard is all red.

Difficulty: Advanced · Know this first: Lesson 05, Lesson 12

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

Two zones, standing by

One zone serving, one zone standing by. I start with one zone and build up to the full standby picture.

Zones: 2Standby: full copyShift: traffic manager or DNS
probeprobereplicationDNSpoints traffic at azoneTraffic managershifts the routeHealth checkerprobes each zonePrimary zoneload balancer, pods,data copyStandby zoneload balancer, pods,data copy
your request
your requesthealthy zonestandby copydead zonerequest · flowingresponse · the way back

    Extra diagram · Second trip

    The failover, minute by minute

    A zone dies in the middle of a request. I follow that request from the first missed probe to the answer it finally gets.

    Dead after: 3 missed probesShift: route or DNS TTLWarm-up: the slow part
    Zone diesmid-requestanswer never comesProbe misses oncestill alive?Probe misses twicegetting worriedThird misszone called deadRoute shiftsor DNS TTL expiresStandby warms upboots, warms itscacheStandby answersyour request, atlast
    the in-flight request
    in-flight requestmissed probestandby answersrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a normal request the work is cheap, and in a failover the detection, the shift, and the warm-up eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.

      No standby. The request fails outright. There is nothing to shift to.

      Try it yourself

      The numbers above are one fixed example. Move the controls below and watch the same failover change. The budget bar, the timings on the diagram, and the status lines all update live. The simulator starts from the clean failover scenario. Switching to a different scenario resets these controls to their defaults, so the two never fight each other.

      How many standby zones sit behind the primary. Zones do not change the failover time for one zone failure. They change how many zone failures the system survives. Zero means no failover at all.

      Three missed probes in a row call the zone dead, so detection is three times the interval. At 1 second, detection is 3,000 ms. At 10 seconds, it is 30,000 ms.

      The shift is 500 ms plus the TTL. At a TTL of 0 the traffic manager flips the route itself. At 300 seconds, clients hold the old answer and the shift is 300,500 ms.

      A warm standby that serves real traffic needs about 1.5 seconds. A cold standby that boots from nothing needs minutes. At 120 seconds, warm-up is 120,000 ms.

      About 5.0 seconds of downtime for this request.

      Detection 3,000 ms + shift 500 ms + warm-up 1,500 ms. Total trip: ~5,130 ms.

      Failover lands in seconds. Users notice a pause, not an outage.

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic four times. First the spare copy, then the fast switchover, then the ways the switchover itself fails, then the second region.

      Basic — A second copy that can take over

      Definition: Redundancy means a second copy of everything that can die. The zone, the data, and the route to it. I run the same code in a second zone with the same data, standing by. When the first zone goes dark, the second one takes the traffic. Failover is the move itself. Detection, shift, warm-up. Three pieces, and each one is a setting I chose on a calm day.

      Types:

      • Active-passive: one zone serves traffic, the other waits. Simple to reason about, one writer, one source of truth.
      • Active-active: both zones serve traffic all the time. Better utilization, harder writes.
      • Warm standby: the standby serves a small slice of real traffic, about 5 percent, so it stays hot and tested.
      • Cold standby: the standby sits idle until the bad day. Cheap, and untested.

      Uses: I use it for the day a whole zone goes dark. I have watched that happen. The request in flight dies with the zone, and the next request has to land somewhere else. I also use it for planned work. A zone drained for maintenance is a failover with a calendar invite instead of a pager.

      Dimensions: Active-passive costs double the machines for the same traffic, and the standby sits near idle. That is the deal. I pay for capacity I hope never to use, the same way I pay for insurance. Failover time is detection plus shift plus warm-up. A cold standby can take three minutes to answer the first request. The trade is money for uptime. The standby earns its keep the one day the primary dies.

      Intermediate — Making the switchover fast

      Definition: A fast failover is three numbers I control. How often the health check probes, how many misses call a zone dead, and how long a DNS answer lives. I probe every second and call the zone dead after three misses, so detection takes about three seconds. I shift traffic at the traffic manager, not through DNS, because a DNS TTL of 300 seconds is a five-minute failover written down in advance.

      Types:

      • Health-check probing: interval and miss threshold. Shorter interval, faster detection, more false alarms.
      • Traffic-manager shift: the route changes at the entry point. Fast, and the entry point itself needs redundancy.
      • DNS shift: simple, slow, and cached everywhere I cannot control.
      • Live-traffic warming: 5 percent of real requests go to the standby every day, so its caches stay hot.

      Uses: I use this when I am setting the failover numbers for a real service and when I am drilling the failover on a schedule. I read the three numbers on a bad day before I trust any dashboard. Dashboards show the failover happened. The config shows how long it was always going to take.

      Dimensions: As load grows, the standby has to absorb the full load of the primary in seconds. If the standby is sized at half the primary, it falls over the moment traffic arrives. I size each zone to carry the full load alone. The failure I watch for is flapping. A health check that is too sensitive flips traffic back and forth, and users get a different answer on every retry. The cost is drill time and 5 percent of traffic. The trade is detection speed against stability. A hold time after a flip lets the system settle before it flips again.

      Advanced — The failures that look like success

      Definition: The failover can work perfectly and the system can still go down. The standby takes over with data that is 40 seconds old, and users see their own recent writes vanish. Or both zones think they are the primary and both take writes, and the writes conflict when the zones meet again. Or the standby has a different config, a different secret, a different version, because the last config change went to the primary and never went to the standby. The failover worked. The system still went down.

      Types:

      • Split brain: two primaries, conflicting writes. The fix is fencing. The old primary must be stopped before the new one starts.
      • Replication lag at promotion: the gap between the primary and the standby is the list of writes I lost. I check the lag before I promote.
      • Config drift: the standby drifted from the primary over months. The fix is one deploy pipeline for every zone.
      • Untested failover: the runbook says 30 seconds. The runbook was written two years ago.

      Uses: I use this in the first five minutes of an incident, when I check the lag reading before I promote and write down the gap. I would rather tell the team the number of lost writes in the first five minutes than discover it from users the next day. I use it in reviews, when I ask when the failover was last drilled. A drill nobody runs catches nothing.

      Dimensions: At ten times the traffic, one standby zone stops being enough for the failover itself, and the traffic manager becomes the thing I watch, because one global entry point is itself a single point of failure. The cost is fencing, lag checks, and a monthly drill in a staging zone that looks like production. The trade is promotion speed against data safety. Promoting fast with stale data is a data loss plan with extra steps. Promotion without fencing is not failover.

      Master — Regions, not zones

      Definition: At this level the failure is not one zone. It is a whole region going dark. When the us-east-1 region had its big outage in December 2021, the services that survived were the ones whose standby was warm and tested, and the ones that went down hardest were the ones whose redundancy stopped at the region boundary. A second region means copies of the data in another region, async replication across regions, and a recovery time measured in minutes instead of seconds.

      Types:

      • Sync in-region, async cross-region: fast replication nearby, slower replication far away, because the speed of light does not negotiate.
      • Region-level steering: traffic moves between regions the same way it moves between zones, with the same health checks.
      • Data residency splits: some data cannot leave its region, so the design has to own that from the start.

      Uses: I use this when one region is a single point of failure for the business, not just a zone. I use it when the recovery target is minutes and the budget allows a second region standing by. And I use it when I have to say plainly that a user reading from the far region is reading the past by design, and the design has to own that.

      Dimensions: Cross-region replication lag is measured in seconds or minutes, not milliseconds, and that lag is the data loss on a region failover. The cost is a whole second region, plus the team that keeps it warm. The failure mode is a region failover that works and still loses the last minute of writes, because async replication had not caught up. The trade is consistency against distance. I keep sync replication inside one region and async across regions, and I write the recovery time down before the bad day, not during it.

      Go deeper

      The lesson above is the shape of a clean failover. Production is messier. Here is what I actually argue about when I set this up for real.

      The trade I keep making

      Active-passive is the simple version. One zone serves traffic. The other waits. It costs double the machines for the same traffic, and the standby sits near idle. That is the deal. I pay for capacity I hope never to use, the same way I pay for insurance. The upside is the standby is simple to reason about. There is one writer, one source of truth, and no fights over who owns a write.

      Active-active is the expensive version of the same idea. Both zones serve traffic all the time. Utilization looks good on the dashboard and nobody asks why a zone sits idle. The cost moves somewhere else. Now two zones take writes, and writes in two places have to agree. That means conflict rules, or a single writer with cross zone latency, or accepting that two users can book the same seat at the same second. I only go active-active when the traffic justifies it and the data can take it. For most systems, active-passive with a warm standby is the right call, and the money saved on complexity pays for the idle machines.

      The middle path is the one I actually use most. The standby serves a small slice of real traffic, about 5 percent. It stays warm, its caches stay hot, and I see its errors in the dashboard every day instead of discovering them during an outage. It costs a little routing work. It saves me from the cold standby surprise.

      Failure modes that actually happen

      Split brain is the one I plan for and hope never to see. Two zones both think they are the primary, and both take writes. When they meet again, the writes conflict and somebody loses. The fix is fencing. The old primary must be stopped from taking writes before the new one starts. Promotion without fencing is not failover. It is a data loss plan with extra steps.

      Replication lag at the moment of failover is quieter and more common. The standby takes over with data that is 40 seconds old. Users see their own recent writes vanish. I check the lag reading before I promote, and I write down the gap. That gap is the list of writes I lost. I would rather tell the team that number in the first five minutes than discover it from users the next day.

      A failover that never got tested is the most common failure of all. The runbook says it takes 30 seconds. The runbook was written two years ago. The standby has a different config now, a different secret, a different version. I test failover on a schedule, not on hope. A monthly drill in a staging zone that looks like production catches the drift. A drill nobody runs catches nothing.

      Flapping is the failure that looks like success. The health check is too sensitive, so the traffic manager flips to the standby, then back to the primary, then to the standby again. Users get a different answer on every retry. I set the miss threshold high enough that a slow probe does not look like a dead zone, and I add a hold time after a flip so the system settles before it flips again.

      What changes at 10x and 100x

      At 10x the traffic, one standby zone stops being enough for the failover itself. The standby has to absorb the full load of the primary in seconds. If the standby is sized at half the primary, it falls over the moment traffic arrives. I size each zone to carry the full load alone, not half of it. That is the same rule as replicas in Lesson 12. Every copy must carry the whole load by itself.

      At 10x the zones, the traffic manager becomes the thing I watch. One global entry point that routes to many zones is itself a single point of failure, so I run it in more than one place too, with health checks on the checkers. Redundancy has to cover the redundancy.

      At 100x, failover stops being a zone problem and becomes a region problem. A whole region can go dark, not just a zone. That means copies of the data in a second region, async replication across regions from Lesson 12, and a recovery time measured in minutes instead of seconds. I keep sync replication inside one region and async across regions, because the speed of light does not negotiate. A user reading from the far region is reading the past by design, and the design has to own that.

      The config I read on a bad day

      A short health check and failover policy, in the shape I actually keep in a runbook. The names are generic. The point is the three numbers that decide the failover time: how often to probe, how many misses call it dead, and how long a DNS answer lives.

      # Health check: probe every 1s, call the zone dead after 3 misses.
      # Detection time is 3s. Shorten the interval and detection gets
      # faster. Shorten it too far and a slow probe looks like a dead zone.
      health_check:
        interval_seconds: 1
        missed_probes_to_fail: 3
      
      # Failover: shift traffic at the traffic manager, not through DNS.
      # A DNS TTL of 300s is a 5 minute failover written down in advance.
      failover:
        mode: traffic_manager
        dns_ttl_seconds: 0
      
      # Standby: keep it warm with 5 percent of real traffic.
      # A standby that never serves a request is a standby you never tested.
      standby:
        live_traffic_percent: 5

      I read those three numbers on a bad day before I trust any dashboard. Dashboards show the failover happened. The config shows how long it was always going to take.

      One time it broke

      A whole zone went dark on a weekday afternoon. The traffic manager noticed, the shift happened, and the standby took over. That part worked. Then every request on the standby started failing. Not slow. Failing. The standby pods were running and healthy. They were calling an internal address that only existed in the dead zone. A config change months earlier had pointed a downstream service at the primary zone's private address. The change went to the primary. It never went to the standby. The standby had never served a real request, so nobody noticed. The failover worked perfectly and the system still went down.

      • 14:02The primary zone goes dark. Requests hang. The pager fires.
      • 14:05Health checks call the zone dead. The traffic manager shifts to the standby.
      • 14:06The standby starts receiving traffic. Every request errors. The dashboard goes red in a new place.
      • 14:12I check the standby pods. They are running. The logs show connection timeouts to an internal address.
      • 14:20I find the cause. The downstream address is the primary zone's private address. It died with the zone. The standby config still points at it.
      • 14:28I push the corrected config to the standby and restart the pods. Requests start succeeding.
      • 14:40Traffic is normal on the standby. The outage lasted 38 minutes. The failover itself took 3 minutes. The config took the other 35.

      Three things changed after, and I still do them now. First, the standby serves 5 percent of real traffic every day, so a wrong config shows up on a quiet Tuesday instead of during an outage. Second, config deploys go to every zone in one pipeline. A change that reaches only one zone is a failed deploy, and the pipeline says so. Third, failover gets drilled once a month in a staging zone that looks like production. The drill is not a demo. It is a test, and it fails until it passes.

      If you remember nothing else

      Try designing it

      Design failover for a checkout API across two zones. The API takes 2,000 requests per second at peak. Recovery time objective is 60 seconds: the API must be answering again within a minute of a zone dying. Recovery point objective is near zero: a completed checkout must never vanish. The checkout writes to a database, and the database already replicates to the standby zone.

      Clarifying questions to ask first

      • What does 60 seconds include: detection, shift, and warm-up, or just the shift? The answer changes every number in the design.
      • Which writes can never be lost? A completed checkout is one answer. A cart update is a different one.
      • How much replication lag is acceptable at the moment of failover? Near zero means sync replication, and sync costs latency on every write.
      • How will you know the standby works? A standby that never serves traffic is a guess, not a plan.

      Worked approach

      • Start with active-passive: one primary zone, one standby zone. The standby runs the same code and the same data, and it serves 5 percent of real traffic so it stays warm.
      • Use sync replication for checkout writes inside the region, so a completed checkout survives a dead primary. That costs roughly 20 to 50 ms per write. Use async for cart updates, where a few seconds of staleness hurts nobody.
      • Set the health check interval to 1 second with 3 missed probes to fail. Detection is 3 seconds. Shift at the traffic manager, not through DNS, so the shift is about 500 ms. Warm-up is about 1.5 seconds because the standby already serves traffic. Total failover is about 5 seconds, well inside the 60 second objective.
      • Size the standby to carry the full 2,000 requests per second alone. If it is sized at half the primary, it falls over the moment traffic arrives, and the failover becomes a second outage.
      • Fence the old primary before promoting the standby. If the old zone comes back and takes writes again, two primaries will disagree, and somebody loses a checkout.
      • Rough sizing: 2 zones, each sized for 2,000 requests per second, so the bill is double the single zone cost. The standby at 5 percent traffic serves about 100 requests per second day to day, which keeps its caches hot for the cost of a small slice of routing work.

      Trade-offs to name

      • Sync replication for checkouts costs latency on every write. Async would be faster, and it would risk losing a checkout the user already saw as placed.
      • Active-passive costs double the machines for the same traffic. Active-active would use the capacity better, but two zones taking checkout writes need conflict rules, and a double charge is worse than an idle server.
      • A 1 second health check interval notices fast, but a slow probe can look like a dead zone. A longer interval is safer against false alarms and slower on the real day. There is no free setting here.
      Saved in this browser only.
      Next in this series
      Lesson 23 — Circuit Breakers and Bulkheads