Platform Engineering · Lesson 21

Getting Machines to Agree

One machine takes the writes. The rest copy it. That works until the one machine dies. Then the cluster has a problem that no amount of retrying will fix. Somebody has to decide who leads next, and everybody has to agree on the answer. This lesson follows one election after the leader dies, hop by hop, the way I trace it when a cluster stops taking writes at 2 AM and the dashboard says every node is up.

Difficulty: Advanced · Know this first: Lesson 12, Lesson 20

Lesson 12 (Keeping Extra Copies That Agree) first, because the election in this lesson is about agreeing on a log, and Lesson 12 is where that log and its copies are explained. Lesson 20 (Pick Two: The CAP Trade) first, because it explains why a minority of nodes cannot decide alone, and this lesson is the mechanism that enforces that rule. Without those two, the quorum math in this lesson looks like an arbitrary rule instead of the thing that keeps the data in one piece.

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

Three nodes, one leader

One leader, two followers, one heartbeat. I start with two nodes and build up to the majority line.

Nodes: 3Majority: 2 of 3Leader: 1 at a time
heartbeatheartbeatvotevotecannot elect aloneNode 1leaderNode 2followerNode 3followerMajority line2 of 3Minority side1 node cannot elect
the heartbeat
heartbeat / voteleaderfollowerrequest · flowingresponse · the way back

    Extra diagram · Second trip

    When the vote splits

    The leader dies, two followers time out together, and the vote splits. I follow the election from the split to the new leader.

    Split vote: 1-1Term: advances each electionMajority still: 2 of 3
    votes for itselfvotes for itselfnew electionstale termLeader diesno more heartbeatsBoth time outat the same momentNode 2 standscandidateNode 3 standscandidateVotes split 1-1no majorityTerm advancesnext termNew leaderwins a majorityOld leader returnsstale term, stepsdown
    the election
    the electionsplit / stalenew leaderrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is honest though. In a clean election I have watched, the waiting and the voting eat most of the time, and the actual work of catching up the log is the rest. Step through the lesson or switch the scenario and watch the bar change.

      Try it: size the election

      Move the controls and watch the election change. This is the clean-election path, the leader-dies scenario, with your settings. The budget bar above and the step timings and status lines update live. The diagram reflects the change too.

      Changes the majority needed, the failures tolerated, and the vote, decision, and log catch-up steps.

      Sets the Timeout fires hop directly. A longer timeout means a slower election and fewer false elections when the network hiccups.

      One follower is silent during the election. Votes requested gains 20 ms, Majority decides gains 10 ms, Log catches up gains 40 ms. Total gains 70 ms.

      Quorum: 2 of 3 nodes must agree. Tolerates 1 failures.

      Election total with your settings: ~550 ms

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic four times. First the idea, then the knobs, then the network trying to break it, then the judgment that keeps a cluster alive at 2 AM.

      Basic — What it is

      Definition: Consensus is how a group of machines agrees on one answer when any of them can die. Leader election is the part of that agreement that picks one machine to take the writes. I run a small coordination store with three nodes. One node is the leader. Only the leader takes writes. The other two copy the leader's log. When the leader dies, the followers hold a vote and pick a new one. That is the whole idea.

      Types:

      • One leader, many followers: a single writer, so every change has one order.
      • Voting: a candidate asks for votes and wins with a majority.
      • Terms: a number that says which election is current. A higher term always wins.

      Uses: I use it anywhere two writers would corrupt the data. A config store that holds feature flags. A lock service that hands out one lock at a time. A registry that maps service names to addresses. If two nodes could take the same write and disagree, I need one leader, not two.

      Dimensions: A three-node cluster keeps working when one node dies, because two out of three is still a majority. It stops taking writes when two die, and that is correct. It is better to stop than to split. An election pauses writes for a few hundred milliseconds while the vote runs. The cost is three machines for one job, and the trade is consistency for availability. I give up taking writes during a minority partition so I never give out two different answers.

      Intermediate — Running the election

      Definition: The election runs on heartbeats and timeouts. The leader sends a heartbeat every 100 ms. A follower that hears nothing for about 1,000 ms decides the leader is gone, bumps the term, and asks for votes. A candidate only wins if its log is at least as fresh as the voter's log. That one check stops a node that was down for a week from coming back and dropping a day of writes.

      Types:

      • Randomized timeouts: each follower waits a different amount of time, so two followers rarely stand at once.
      • Pre-vote: a follower asks first, would I win if I stood, and only stands if the answer is yes.
      • Log freshness check: a voter refuses a candidate whose log is behind its own.

      Uses: I use this when I am the one tuning a real cluster, not just reading about one. I set the heartbeat and the timeout from measured network delay, not from a default. On call, I read the leader-change count on the dashboard. A healthy cluster changes leader maybe once a month. Forty-one changes in an hour means the timeout is wrong, not the cluster.

      Dimensions: As the cluster grows to a million keys, the vote itself stays fast but the catch-up hop grows, because the new leader has to check and copy a million keys of log. The failure I watch for is flapping. A timeout of 150 ms on a network that regularly delays a heartbeat by 200 ms elects a new leader every few seconds. The cost is tuning time and one dashboard panel. The trade is speed against false alarms. A short timeout elects faster and flaps more. A long timeout flaps less and leaves writes waiting longer.

      Advanced — Surviving the network

      Definition: The hard case is not a dead leader. It is a live leader the followers cannot hear. A network partition cuts the cluster in two. Each side thinks the other side died. The side with the majority elects a new leader and keeps writing. The side without a majority stops. That is the rule that keeps the data in one piece. When the partition heals, the old leader sees a higher term and steps down.

      Types:

      • Fencing by term: a write stamped with an old term is refused, so a stale leader cannot write after it lost.
      • Quorum reads: a read that asks a majority, for the cases where a follower's copy is not fresh enough.
      • Lease-based leadership: the leader holds a lease that expires, so a partitioned leader stops acting when the lease runs out.

      Uses: I use this when the cluster runs across zones and the network between them is a real failure mode, not a theory. I use it when an old leader comes back still thinking it leads, the way the split-vote scenario in this lesson shows. And I use it in design reviews, when someone proposes two writers to get more throughput. My answer is no. Two writers is how data gets written twice in two places and lost in both.

      Dimensions: At ten times the keys, a bad election costs ten times as much catch-up time, so the timeout numbers matter more, not less. The breakage I notice first is writes freezing for half a second at a time, over and over, while every node reports healthy. The cost is the discipline to test partitions on purpose, not just node deaths. The trade never changes. Safety means the minority goes quiet. Availability means the minority keeps serving and the data splits. I pick safety every time.

      Master — Designing the cluster

      Definition: At this level I am not tuning one cluster. I am deciding how many clusters there are, how big each one is, and where the nodes live. Five nodes tolerate two failures and need three votes to write. Seven tolerate three and need four. Every extra node buys tolerance and charges write latency, because each write waits for more followers. At a hundred times the scale I stop running one big cluster and shard the leadership instead, one small elected group per shard, so one election never stops every write in the system.

      Types:

      • Three nodes: the default. Tolerates one failure, writes wait for one follower.
      • Five nodes: tolerates two failures, writes wait for two followers. The smallest cluster that keeps writing with two nodes down.
      • Sharded leadership: one small elected group per shard, so a single election has a small blast radius.
      • Witness nodes: a node that votes but holds no data, for breaking ties cheaply.

      Uses: I use this when I am designing the coordination layer for a whole platform, not one service. I put the five nodes across three zones, never three in one zone, so one zone failure cannot take the majority. I set the alert on leader changes per hour before the cluster ever serves traffic. And I write the runbook line that says what to do when the cluster will not elect: check the term numbers first, then the log freshness, then the timeouts.

      Dimensions: A five-node cluster costs more than three and every write is slower, because it waits for two followers instead of one. Reads from followers can be stale by a heartbeat, so a client that just wrote a flag reads from the leader if it needs to see its own write. The failure mode is a slow election across a big cluster, where catch-up dominates and the vote is the cheap part. The trade is tolerance against speed and cost. I do not jump to seven nodes unless the requirement says three failures, because each extra node is slower writes and a longer election for tolerance I may never need.

      Go deeper

      The election above is the simple case. Production clusters add three things, and each one exists because somebody got hurt without it. First, pre-vote. A follower that has been cut off from the cluster can keep bumping the term every time its timeout fires, and each bump forces a real election when it rejoins. Pre-vote lets it ask first, would I win if I stood, and only stand if the answer is yes. That one check stops a returning node from wrecking a healthy leader. Second, the log freshness rule. A candidate only wins votes if its log is at least as fresh as the voter's. I have seen a node that was down for a week come back, win an election on a stale log, and drop a day of writes that were never copied to a majority. The freshness check is the guard against that. Third, the timeout numbers themselves. A common production default is a 100 ms heartbeat and a 1,000 ms election timeout, roughly ten heartbeats of silence before anyone acts. That ratio is not magic. It is ten chances for the network to recover before the cluster pays for an election. At 10x scale, the timeouts do not change much, but the cost of a bad election does. A leader change in a cluster with a million keys means a million keys of log to check and copy, so the catch-up hop grows and the election feels longer even though the vote itself is the same speed. At 100x scale, I stop running one big cluster and I shard the leadership instead, one small elected group per shard, so one election never stops every write in the system. The trade never changes. Fast elections risk false alarms. Slow elections risk long outages. I pick the timeout that matches how flaky my network actually is, and I measure leader changes per day to prove the pick was right.

      The snippet below is an example config for a three node coordination store. It is not copied from any vendor doc. It shows the two numbers that matter most.

      # three node coordination store, example config
      name: node-1
      heartbeat-interval: 100   # ms between leader heartbeats
      election-timeout: 1000     # ms of silence before a follower stands
      # ten heartbeats of silence before anyone acts.
      # a shorter timeout elects faster and flaps more.
      # a longer timeout flaps less and leaves writes waiting longer.

      In our lesson the timeout is 180 ms so the election fits on one screen. In production it is usually closer to 1,000 ms, because a real network hiccups more than a diagram does.

      One time it broke

      A three node coordination cluster that held service registry entries started electing a new leader every few seconds. Each election froze writes for about half a second. Services kept re-registering, health checks flapped, and the on-call phone went off four times in one hour. The data was safe. Nothing was lost. But nothing could write for long enough to matter, and the registry went stale while the cluster argued with itself.

      • 02:14Pager fires. Registry writes timing out. Dashboard shows all three nodes up and healthy.
      • 02:20I pull the leader change count. The cluster has changed leader 41 times in the last hour. A healthy cluster changes leader maybe once a month.
      • 02:31I compare the timeout settings. Election timeout is 150 ms on a network where a busy hour regularly delays a heartbeat by 200 ms. The followers are not seeing a dead leader. They are seeing a slow network and calling it death.
      • 02:40I raise the election timeout to 1,000 ms and the heartbeat interval stays at 100 ms. Leader changes stop within a minute.
      • 03:05Writes recover. The stale registry entries clear as services re-register against the stable leader. I stay on the bridge until the leader has held for thirty minutes straight.

      What changed after

      Three things. The timeout is now set from measured heartbeat delay, not from a default, and it is reviewed whenever the network changes. The leader change count is a dashboard panel with an alert, so the next flapping cluster pages us on the second election, not the fortieth. And the three nodes now sit in three separate racks, so one rack problem cannot take two nodes and leave a single follower that can never reach a majority.

      If you remember nothing else

      Design exercise

      Design leader election for a five node config store. It holds feature flags and service addresses. Reads are 10,000 per second. Writes are rare, maybe ten per second, but a lost write means a service keeps running with the wrong flag. The store must keep taking writes when two nodes are down. It must never take writes from two leaders at once. Sketch the election, the quorum, and what happens when the leader dies.

      Clarifying questions to ask

      • How many nodes can be down at once and still take writes?
      • What is the read load, and can reads be served by followers?
      • How long can writes pause during an election before clients start failing?
      • Where do the five nodes run, and can one rack or zone take out more than two of them?

      Worked approach

      Start with five nodes, because five tolerates two failures and three is the majority. That is the smallest cluster that meets the requirement, and I would say so out loud. Writes go only to the leader, and the leader copies each write to a majority, three nodes, before it answers. Reads go to any node, followers included, because reads at 10,000 per second do not need the leader. When the leader dies, the followers wait a random timeout, the first one stands, and it needs three votes to win. Two nodes down still leaves three up, which is exactly a majority, so writes continue. Three nodes down leaves two, which is not a majority, so the cluster stops taking writes and waits. That is correct. It is better to stop than to split. Put the five nodes across three zones, never three in one zone, so one zone failure cannot take the majority. Set the heartbeat to 100 ms and the election timeout to 1,000 ms, and alert on leader changes per hour. If the interviewer pushes for faster elections, I say the timeout can drop, but every drop buys faster recovery with more false elections, and I would measure heartbeat delay first before promising a number.

      Trade-offs to name

      Five nodes cost more than three and every write waits for two followers instead of one, so writes are slower. Reads from followers can be stale by a heartbeat, so a client that just wrote a flag must read from the leader if it needs to see its own write. A bigger cluster tolerates more failures but elects slower and copies to more nodes, so I would not jump to seven nodes unless the requirement said three failures.

      Next in this series
      Lesson 22 — When One Fails, Another Steps In