Platform Engineering · Lesson 20

Pick Two: The CAP Trade

Three copies of the same data should agree. Most days they do. Then the network between them is cut, and I have to choose. I can keep taking writes and let the copies disagree, or I can keep the copies in agreement and refuse writes. I cannot do both. This lesson follows one write through a three replica store while the network is cut, hop by hop, the way I trace it when a user says their change vanished.

Difficulty: Advanced · Know this first: Lesson 09, Lesson 12

Lesson 09 (Tables or Documents) first, because you need to know what kind of store you are copying before the copies and the consistency rules mean anything. Lesson 12 (Keeping Extra Copies That Agree) first, because this lesson is replication under a partition, and Lesson 12 is where the copies and the lag between them are explained. Without those two, the quorum hops in this lesson look like magic.

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

Three copies, one cut

One write, three copies, and a cut through the middle. I start with one copy and build up to the split the lesson is about.

Copies: 3Write quorum: 2 of 3Read quorum: 2 of 3
read paththe cutAppwrites and readsCoordinatorsends the write outReplica 1copy AReplica 2copy BReplica 3copy C
the write
the writethe cutcopies agreestale copyrequest · flowingresponse · the way back

    Extra diagram · Second trip

    After the cut heals: the repair trip

    The cut closes. I follow the newest value as it repairs the stale copies, until every copy agrees again.

    Tie-breaker: version numberCopies: 3Reads after repair: any copy
    Partition healsthe cut closesCompare versionswhich copy is newestNewest valueversion winsStale copies updatedcopy the newestvalueAll three agreesame versioneverywhereNext readtrusts any copy
    the newest value
    the newest valueall agreestale until updatedrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a normal write the copies agree in milliseconds, and in a partition the two failed copy attempts eat most of the budget before the choice is even made. Step through the lesson or switch the scenario and watch the bar change.

      Try it yourself

      The numbers above are one fixed example. Move the controls below and watch the same write change. The budget bar, the timings on the diagram, and the status lines all update live. The simulator starts from the normal scenario. Switching to a different scenario resets these controls to their defaults, so the two never fight each other.

      3

      How many copies hold the value. More replicas means more copies to wait for at higher quorums, and more copies that can disagree during a partition. It also changes the partition math: the minority side of a cut holds fewer replicas, so a high write quorum cannot be met there.

      3

      How many replicas must ack before the write returns. A higher write quorum waits longer and survives less of a partition. A lower one answers faster and risks a stale read. Write wait time = 8 ms local commit + (W − 1) × 12 ms.

      2

      How many replicas must answer a read before the coordinator returns a value. A higher read quorum waits longer and is more likely to meet a replica that saw the newest write. Step 8 takes 8 ms + (R − 1) × 7 ms.

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The lesson above follows one write through a three replica store while the network is cut. This section covers the same write four times, from the first choice between two of three to the level where strong and eventual live in different stores on purpose.

      Basic — Two of three, and the partition picks

      Definition: Three copies of the same data should agree. Most days they do. Then the network between them is cut, and I have to choose. I can keep taking writes and let the copies disagree, or I can keep the copies in agreement and refuse writes. I cannot do both. The cut makes the choice for me, and the only choice left is which side I stand on.

      Types:

      • Consistency — every read sees the newest write, no matter which copy answers.
      • Availability — every request gets an answer, even when the copies disagree.
      • Partition tolerance — the system keeps running when the network between the copies is cut.
      • The replica — one of the three copies holding the value.

      Uses: I use this level when someone quotes CAP to justify a slow system on a healthy day. CAP is about partitions, not everyday uptime. When the network is healthy, I can have consistency and availability together, and I do. A slow system on a healthy day is not CAP. It is a slow system.

      Dimensions:

      • Load: At three replicas and no partition, the copies agree in milliseconds. The choice costs nothing, because there is no choice to make. The theorem only bites when the network is cut.
      • Breaks: It breaks silently. A rotation change lands on one replica, the app shows saved, and the read that routes the page hits two replicas that never got it. I notice it when the page goes to the wrong person and nothing has failed or errored.
      • Cost: The cost of the choice is paid by someone. Consistency costs the user who sees an error during a partition. Availability costs the user who sees their own change vanish.
      • Trade-off: I give up one of the three to get the other two. The partition picks which two are on the table. My job is to decide, before the cut, in config, which side I stand on when it happens.

      Intermediate — Quorums buy back strong reads

      Definition: Quorums are how I buy back consistency without waiting for every replica. W plus R greater than N is the rule I check first. If the write quorum and the read quorum always overlap, a read always meets at least one replica that saw the newest write. That overlap is the whole trick. I do not need all replicas to agree. I need the two quorums to meet.

      Types:

      • N — how many copies hold the value, 3 in this lesson.
      • W — how many replicas must ack before the write returns, 2 in the lesson's worked numbers.
      • R — how many replicas must answer a read before the coordinator returns a value, 2 here.
      • The overlap rule — W plus R greater than N, so 2 plus 2 is 4, greater than 3, and a read quorum always meets a write quorum.

      Uses: I use this level when I set the three numbers. I read them on a bad day before I trust any dashboard. Dashboards show lag. The config shows what the ack actually promised. Drop W to 1 and the sum is 3, and reads can miss the write entirely. A write that waits for 2 acks takes about 20 ms in this lesson's numbers, and it survives one dead replica.

      Dimensions:

      • Load: At higher quorums, every write waits for more acks and every read asks more replicas. A write wait is 8 ms for the local commit plus 12 ms for each extra ack, so W of 3 costs more than W of 2, and the cost lands on every write.
      • Breaks: It breaks when a quorum waits for the slowest replica. A slow replica acks, but late, and one slow machine turns into slow writes for everyone. I notice it when the quorum timeout fires often. The fix is to fix or remove the slow replica, not to raise the timeout.
      • Cost: The cost is latency on every write and every read, compared to W of 1 and R of 1. The fast pair would answer in about half the time, and it would let stale reads through by design.
      • Trade-off: I give up the fastest read and the fastest write to get a read that cannot miss the newest write. Which side carries the weight changes with the traffic, but the sum still has to clear N.

      Advanced — Eventual, monotonic, and repair

      Definition: Eventual consistency is not a shrug. It is a promise with a time attached, and I have to name that time. If the copies agree in 50 ms, most users never notice. If they agree in 5 minutes, users notice every time. Monotonic reads are the smaller promise I make when I cannot afford the big one. A user must never see a newer value and then an older one on the next reload. That flip flop is what makes users distrust the page.

      Types:

      • Eventual with a number — the expected time for the copies to agree, written down and alerted on when it is missed.
      • Monotonic reads — the next read is at least as fresh as the last one, kept by pinning the user to one replica or carrying the version in the session.
      • Read repair — when a quorum read finds replicas that disagree, the coordinator sends the newest value back to the stale ones.
      • Anti-entropy — the background job that walks the data and compares versions, so values nobody reads still converge.

      Uses: I use this level when a user says their change vanished. I route a read right after a write to a quorum, not to a single replica. That one routing rule fixes the vanishing change problem from this lesson. Reads from other users can go to one replica, because a few seconds of staleness hurts nobody there. And I run both repairs. Read repair fixes what users touch. Anti-entropy fixes the rest.

      Dimensions:

      • Load: At 2,000 reads a second across 3 replicas, each carries about 670 a second. If one replica dies, the surviving two take 1,000 each, so each replica must be sized to carry at least half the read load alone, not just a third of it.
      • Breaks: It breaks when a partial partition makes both sides think the other is dead. Both keep taking writes. When the network heals, two versions of the same value meet, and somebody has to lose. I notice it when the copies agree and a write is gone.
      • Cost: The cost is the repair work itself. Read repair is free when reads happen and useless when they do not. Anti-entropy walks the whole store on a schedule, and that walk is load the users never asked for.
      • Trade-off: I give up every read being fresh to get reads that are fast. A read from one replica is quick and can be stale. A quorum read is fresh and costs the wait. I pick per read, not per system.

      Master — Conflicts, clocks, and splitting the store

      Definition: At this level I stop pretending one store fits every write. Strong consistency lives in a small store for the writes I cannot lose. Eventual consistency lives in a big store for the reads that can be stale. The two stores sync async, and the boundary between them is written down. And when two writes land on different sides of a partition, I have already decided what wins, because deciding after the fact is how data gets lost.

      Types:

      • Last write wins — simple, and it loses writes, so I use it only where losing one change is acceptable.
      • Kept conflicts — both versions survive and a human or a merge rule sorts them out, for fields like a delivery address where losing a change is worse than asking once.
      • Version numbers — the ordering I trust, because I do not trust wall clocks for ordering.
      • Region-local quorums — a quorum inside one region, async replication across regions, so a user in another region reads the past by design.

      Uses: I use this level when a clock decides a conflict. Two writes land at nearly the same time, and the one with the later clock wins, even if it happened first. I have seen a clock 30 seconds fast decide a conflict, and the wrong value won. I trust version numbers and vector clocks, or I accept that the loser is chosen by a clock I do not control. I also say which region holds the truth for each kind of data, in the runbook, before anyone asks.

      Dimensions:

      • Load: At 10x the regions, the speed of light joins the argument. A quorum across regions costs a hundred milliseconds or more, because the ack has to cross the country. At 100x, one store cannot carry every write strongly, and the split is what keeps the system standing.
      • Breaks: It breaks when a silent partition stays silent until a user notices. I notice it when a rotation change is fine on one replica and wrong everywhere the page actually reads. The link between the replica groups gets its own alert now, because a silent partition is a stale read waiting to happen.
      • Cost: The cost is two stores, a sync between them, and the runbook that says where the truth lives for each kind of data. A system that tries to make every write strong at 100x pays quorum latency on reads that never needed it.
      • Trade-off: I give up one store with one answer to get a system where each write pays only for the consistency it needs. The strong store is small and slow. The big store is fast and stale. The boundary between them is the design.

      Go deeper

      The lesson above is the clean shape of CAP. Production is messier. Here is what I actually argue about when I set this up for real.

      The trade I keep making

      CAP is about partitions, not everyday uptime. When the network is healthy, I can have consistency and availability together, and I do. The theorem only bites when the network is cut. That is the first thing I say when someone quotes CAP to justify a slow system on a healthy day. A slow system on a healthy day is not CAP. It is a slow system.

      Quorums are how I buy back consistency without waiting for every replica. W + R greater than N is the rule I check first. If the write quorum and the read quorum always overlap, a read always meets at least one replica that saw the newest write. That overlap is the whole trick. I do not need all replicas to agree. I need the two quorums to meet.

      Eventual consistency is not a shrug. It is a promise with a time attached, and I have to name that time. If the copies agree in 50 ms, most users never notice. If they agree in 5 minutes, users notice every time. I write the expected time down, and I alert when the copies stay apart longer than that. Eventual without a number is just stale with better marketing.

      Monotonic reads are the smaller promise I make when I cannot afford the big one. A user must never see a newer value and then an older one on the next reload. That flip flop is what makes users distrust the page. I keep a user on the same replica for a short window after a write, or I carry the version number in the session, so the next read is at least as fresh as the last one.

      Failure modes that actually happen

      A partial partition is worse than a clean cut. One link drops packets, another is slow, and the replicas disagree about who can reach whom. Replica 1 thinks replica 2 is dead. Replica 2 thinks replica 1 is dead. Both keep taking writes. When the network heals, two versions of the same value meet, and somebody has to lose. I use version numbers and last write wins only when losing a write is acceptable. For the rest, I keep the conflict and let a human or a merge rule sort it out.

      A slow replica is quieter than a dead one. It acks, but late. A quorum that waits for the slowest replica turns one slow machine into slow writes for everyone. I set a timeout on quorum waits, and I count how often it fires. If it fires often, the fix is to fix or remove the slow replica, not to raise the timeout.

      Read repair is the quiet fix I rely on. When a read quorum finds replicas that disagree, the coordinator sends the newest value back to the stale ones. The next read is fresh, and nobody had to run a repair job. It only works if reads actually happen. A value nobody reads stays stale until an anti entropy job walks the data and compares versions in the background. I run both. Read repair fixes what users touch. Anti entropy fixes the rest.

      Clock skew breaks last write wins. Two writes land at nearly the same time, and the one with the later clock wins, even if it happened first. I have seen a clock 30 seconds fast decide a conflict, and the wrong value won. I do not trust wall clocks for ordering. I trust version numbers and vector clocks, or I accept that the loser is chosen by a clock I do not control.

      What changes at 10x and 100x

      At 10x the replicas, quorum math gets expensive. Every write waits for more acks, and every read asks more replicas. I stop adding replicas for consistency and start adding them for read scale, with a lower read quorum and a higher write quorum, or the other way around, depending on which side is busier. The sum W + R still has to clear N, but which side carries the weight changes with the traffic.

      At 10x the regions, the speed of light joins the argument. A quorum across regions costs a hundred milliseconds or more, because the ack has to cross the country. I keep quorums inside one region and replicate across regions async. A user in another region reads the past by design, and the design has to own that. I say so in the runbook, and I say which region holds the truth for each kind of data.

      At 100x, I stop pretending one store fits every write. Strong consistency lives in a small store for the writes I cannot lose. Eventual consistency lives in a big store for the reads that can be stale. The two stores sync async, and the boundary between them is written down. A system that tries to make every write strong at 100x is a system that pays quorum latency on reads that never needed it.

      Even without a partition, there is a second trade, and it has a name I use in design reviews. When the network is healthy, I still choose between latency and consistency. A strong read that waits for a quorum is slower than a fast read from one replica. That choice is real on a normal Tuesday, and CAP does not cover it. I name the choice, I measure the latency it costs, and I let the product decide if the freshness is worth it.

      The config I read on a bad day

      A short quorum config, in the shape I actually keep in a runbook. The names are generic. The point is the three numbers and the one rule that ties them together.

      # Replicas in this store.
      N = 3
      
      # Write quorum: how many acks before the write returns.
      # Read quorum: how many replies before the read returns.
      W = 2
      R = 2
      
      # Strong reads only if W + R > N.
      # Here 2 + 2 = 4 > 3, so a read quorum always meets a write quorum.
      # Drop W to 1 and the sum is 3. Reads can miss the write entirely.

      I read those three numbers on a bad day before I trust any dashboard. Dashboards show lag. The config shows what the ack actually promised.

      One time it broke

      A handover changed who was on call. The outgoing engineer updated the rotation in the on call tool, saw saved, and went to bed. An hour later a real alert fired. It paged the outgoing engineer, not the new one. The new engineer never got the page. The outgoing engineer woke up to a phone that should have been quiet, and the alert waited 40 minutes for someone who was not supposed to be on call. The write was fine. It landed on one replica. The read that routed the page hit two replicas that had not got it yet, because the network between the groups was cut and the tool chose availability. Nothing had failed. Nothing had errored. The page just went to the wrong person.

      What changed after: Three things, and I still do them now. First, rotation changes and paging reads use a quorum that overlaps, W = 2 and R = 2 on 3 replicas, so a paging read cannot miss the newest rotation. Second, a rotation change is confirmed by reading it back from a different replica before the tool shows saved. If the read back disagrees, the tool says so instead of saying saved. Third, the link between the replica groups has its own alert now. A cut network used to be silent until a user noticed. It pages now, because a silent partition is a stale read waiting to happen.

      If you remember nothing else

      Try designing it

      Design a user profile store for a small app. The store holds 100,000 user profiles. Each profile has a name, an email, a delivery address, and a notification setting. The app takes 200 profile writes per second at peak and 2,000 profile reads per second at peak. A user must see their own change right after they save it. Other users reading a profile can see a value a few seconds old and nobody will notice. The store runs on 3 replicas in one region, and the network between them can be cut.

      Clarifying questions to ask first

      A worked approach

      Trade-offs to name

      Stored only in this browser, on this device.

      Next in this series
      Lesson 21 — Getting Machines to Agree