Three copies of the same data should agree. Most days they do. Then the network between them is cut, and I have to choose. I can keep taking writes and let the copies disagree, or I can keep the copies in agreement and refuse writes. I cannot do both. This lesson follows one write through a three replica store while the network is cut, hop by hop, the way I trace it when a user says their change vanished.
Difficulty: Advanced · Know this first: Lesson 09, Lesson 12
Lesson 09 (Tables or Documents) first, because you need to know what kind of store you are copying before the copies and the consistency rules mean anything. Lesson 12 (Keeping Extra Copies That Agree) first, because this lesson is replication under a partition, and Lesson 12 is where the copies and the lag between them are explained. Without those two, the quorum hops in this lesson look like magic.
Extra diagram · Architecture map
One write, three copies, and a cut through the middle. I start with one copy and build up to the split the lesson is about.
Extra diagram · Second trip
The cut closes. I follow the newest value as it repairs the stale copies, until every copy agrees again.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a normal write the copies agree in milliseconds, and in a partition the two failed copy attempts eat most of the budget before the choice is even made. Step through the lesson or switch the scenario and watch the bar change.
The numbers above are one fixed example. Move the controls below and watch the same write change. The budget bar, the timings on the diagram, and the status lines all update live. The simulator starts from the normal scenario. Switching to a different scenario resets these controls to their defaults, so the two never fight each other.
How many copies hold the value. More replicas means more copies to wait for at higher quorums, and more copies that can disagree during a partition. It also changes the partition math: the minority side of a cut holds fewer replicas, so a high write quorum cannot be met there.
How many replicas must ack before the write returns. A higher write quorum waits longer and survives less of a partition. A lower one answers faster and risks a stale read. Write wait time = 8 ms local commit + (W − 1) × 12 ms.
How many replicas must answer a read before the coordinator returns a value. A higher read quorum waits longer and is more likely to meet a replica that saw the newest write. Step 8 takes 8 ms + (R − 1) × 7 ms.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above follows one write through a three replica store while the network is cut. This section covers the same write four times, from the first choice between two of three to the level where strong and eventual live in different stores on purpose.
Definition: Three copies of the same data should agree. Most days they do. Then the network between them is cut, and I have to choose. I can keep taking writes and let the copies disagree, or I can keep the copies in agreement and refuse writes. I cannot do both. The cut makes the choice for me, and the only choice left is which side I stand on.
Types:
Uses: I use this level when someone quotes CAP to justify a slow system on a healthy day. CAP is about partitions, not everyday uptime. When the network is healthy, I can have consistency and availability together, and I do. A slow system on a healthy day is not CAP. It is a slow system.
Dimensions:
Definition: Quorums are how I buy back consistency without waiting for every replica. W plus R greater than N is the rule I check first. If the write quorum and the read quorum always overlap, a read always meets at least one replica that saw the newest write. That overlap is the whole trick. I do not need all replicas to agree. I need the two quorums to meet.
Types:
Uses: I use this level when I set the three numbers. I read them on a bad day before I trust any dashboard. Dashboards show lag. The config shows what the ack actually promised. Drop W to 1 and the sum is 3, and reads can miss the write entirely. A write that waits for 2 acks takes about 20 ms in this lesson's numbers, and it survives one dead replica.
Dimensions:
Definition: Eventual consistency is not a shrug. It is a promise with a time attached, and I have to name that time. If the copies agree in 50 ms, most users never notice. If they agree in 5 minutes, users notice every time. Monotonic reads are the smaller promise I make when I cannot afford the big one. A user must never see a newer value and then an older one on the next reload. That flip flop is what makes users distrust the page.
Types:
Uses: I use this level when a user says their change vanished. I route a read right after a write to a quorum, not to a single replica. That one routing rule fixes the vanishing change problem from this lesson. Reads from other users can go to one replica, because a few seconds of staleness hurts nobody there. And I run both repairs. Read repair fixes what users touch. Anti-entropy fixes the rest.
Dimensions:
Definition: At this level I stop pretending one store fits every write. Strong consistency lives in a small store for the writes I cannot lose. Eventual consistency lives in a big store for the reads that can be stale. The two stores sync async, and the boundary between them is written down. And when two writes land on different sides of a partition, I have already decided what wins, because deciding after the fact is how data gets lost.
Types:
Uses: I use this level when a clock decides a conflict. Two writes land at nearly the same time, and the one with the later clock wins, even if it happened first. I have seen a clock 30 seconds fast decide a conflict, and the wrong value won. I trust version numbers and vector clocks, or I accept that the loser is chosen by a clock I do not control. I also say which region holds the truth for each kind of data, in the runbook, before anyone asks.
Dimensions:
The lesson above is the clean shape of CAP. Production is messier. Here is what I actually argue about when I set this up for real.
CAP is about partitions, not everyday uptime. When the network is healthy, I can have consistency and availability together, and I do. The theorem only bites when the network is cut. That is the first thing I say when someone quotes CAP to justify a slow system on a healthy day. A slow system on a healthy day is not CAP. It is a slow system.
Quorums are how I buy back consistency without waiting for every replica. W + R greater than N is the rule I check first. If the write quorum and the read quorum always overlap, a read always meets at least one replica that saw the newest write. That overlap is the whole trick. I do not need all replicas to agree. I need the two quorums to meet.
Eventual consistency is not a shrug. It is a promise with a time attached, and I have to name that time. If the copies agree in 50 ms, most users never notice. If they agree in 5 minutes, users notice every time. I write the expected time down, and I alert when the copies stay apart longer than that. Eventual without a number is just stale with better marketing.
Monotonic reads are the smaller promise I make when I cannot afford the big one. A user must never see a newer value and then an older one on the next reload. That flip flop is what makes users distrust the page. I keep a user on the same replica for a short window after a write, or I carry the version number in the session, so the next read is at least as fresh as the last one.
A partial partition is worse than a clean cut. One link drops packets, another is slow, and the replicas disagree about who can reach whom. Replica 1 thinks replica 2 is dead. Replica 2 thinks replica 1 is dead. Both keep taking writes. When the network heals, two versions of the same value meet, and somebody has to lose. I use version numbers and last write wins only when losing a write is acceptable. For the rest, I keep the conflict and let a human or a merge rule sort it out.
A slow replica is quieter than a dead one. It acks, but late. A quorum that waits for the slowest replica turns one slow machine into slow writes for everyone. I set a timeout on quorum waits, and I count how often it fires. If it fires often, the fix is to fix or remove the slow replica, not to raise the timeout.
Read repair is the quiet fix I rely on. When a read quorum finds replicas that disagree, the coordinator sends the newest value back to the stale ones. The next read is fresh, and nobody had to run a repair job. It only works if reads actually happen. A value nobody reads stays stale until an anti entropy job walks the data and compares versions in the background. I run both. Read repair fixes what users touch. Anti entropy fixes the rest.
Clock skew breaks last write wins. Two writes land at nearly the same time, and the one with the later clock wins, even if it happened first. I have seen a clock 30 seconds fast decide a conflict, and the wrong value won. I do not trust wall clocks for ordering. I trust version numbers and vector clocks, or I accept that the loser is chosen by a clock I do not control.
At 10x the replicas, quorum math gets expensive. Every write waits for more acks, and every read asks more replicas. I stop adding replicas for consistency and start adding them for read scale, with a lower read quorum and a higher write quorum, or the other way around, depending on which side is busier. The sum W + R still has to clear N, but which side carries the weight changes with the traffic.
At 10x the regions, the speed of light joins the argument. A quorum across regions costs a hundred milliseconds or more, because the ack has to cross the country. I keep quorums inside one region and replicate across regions async. A user in another region reads the past by design, and the design has to own that. I say so in the runbook, and I say which region holds the truth for each kind of data.
At 100x, I stop pretending one store fits every write. Strong consistency lives in a small store for the writes I cannot lose. Eventual consistency lives in a big store for the reads that can be stale. The two stores sync async, and the boundary between them is written down. A system that tries to make every write strong at 100x is a system that pays quorum latency on reads that never needed it.
Even without a partition, there is a second trade, and it has a name I use in design reviews. When the network is healthy, I still choose between latency and consistency. A strong read that waits for a quorum is slower than a fast read from one replica. That choice is real on a normal Tuesday, and CAP does not cover it. I name the choice, I measure the latency it costs, and I let the product decide if the freshness is worth it.
A short quorum config, in the shape I actually keep in a runbook. The names are generic. The point is the three numbers and the one rule that ties them together.
# Replicas in this store. N = 3 # Write quorum: how many acks before the write returns. # Read quorum: how many replies before the read returns. W = 2 R = 2 # Strong reads only if W + R > N. # Here 2 + 2 = 4 > 3, so a read quorum always meets a write quorum. # Drop W to 1 and the sum is 3. Reads can miss the write entirely.
I read those three numbers on a bad day before I trust any dashboard. Dashboards show lag. The config shows what the ack actually promised.
A handover changed who was on call. The outgoing engineer updated the rotation in the on call tool, saw saved, and went to bed. An hour later a real alert fired. It paged the outgoing engineer, not the new one. The new engineer never got the page. The outgoing engineer woke up to a phone that should have been quiet, and the alert waited 40 minutes for someone who was not supposed to be on call. The write was fine. It landed on one replica. The read that routed the page hit two replicas that had not got it yet, because the network between the groups was cut and the tool chose availability. Nothing had failed. Nothing had errored. The page just went to the wrong person.
What changed after: Three things, and I still do them now. First, rotation changes and paging reads use a quorum that overlaps, W = 2 and R = 2 on 3 replicas, so a paging read cannot miss the newest rotation. Second, a rotation change is confirmed by reading it back from a different replica before the tool shows saved. If the read back disagrees, the tool says so instead of saying saved. Third, the link between the replica groups has its own alert now. A cut network used to be silent until a user noticed. It pages now, because a silent partition is a stale read waiting to happen.
Design a user profile store for a small app. The store holds 100,000 user profiles. Each profile has a name, an email, a delivery address, and a notification setting. The app takes 200 profile writes per second at peak and 2,000 profile reads per second at peak. A user must see their own change right after they save it. Other users reading a profile can see a value a few seconds old and nobody will notice. The store runs on 3 replicas in one region, and the network between them can be cut.
Stored only in this browser, on this device.