System Design · Lesson 12

Replication

One database is a single point of failure. I have watched a primary die at 2 AM and take every write with it. Replication is the fix: the primary keeps a copy of the data on other machines called replicas. This lesson follows one write from the primary to its replicas, hop by hop, the way I trace it when a user says their update vanished.

Difficulty: Intermediate · Know this first: Lesson 09, Lesson 10

Lesson 09 (SQL vs NoSQL) first, because you need to know what kind of database you are copying before the copies mean anything. Lesson 10 (Storage Engines) first, because replication in this lesson replays the write-ahead log, and Lesson 10 is where that log is explained. Without those two, the WAL hops in this lesson look like magic.

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a normal write the replica catches up in milliseconds, and in a bad one a single hop eats almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.

Try it yourself

The numbers above are one fixed example. Move the controls below and watch the same write change. The budget bar, the timings on the diagram, and the status lines all update live. The simulator starts from the normal scenario. Switching to a different scenario resets these controls to their defaults, so the two never fight each other.

2 replicas

How many copies sit behind the primary. In async mode the count does not change the write time: the primary ships once to the stream and the replicas pull in parallel. In sync mode it changes the wait, because the primary waits for more confirmations.

2 ms

How far behind the replica is. This sets the apply hop (Replica applies it) and the lag reading. A bigger lag means the replica serves older data for longer.

Off means async: the primary ships the change and answers right away. On means sync: the primary waits for a replica to confirm before the ack goes back. At 2 replicas, sync shipping takes 53 ms instead of 3 ms.

Copies agree within a few milliseconds. A read from any replica returns the new value.

The write waits for a replica before the ack. Slower, and the ack now means two copies have it.

One replica means one spare copy at most. If the primary dies before the change ships, there is nothing to promote.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is the happy shape of replication. Production is messier. Here is what I actually argue about when I set this up for real.

The trade I keep making

Async is the default for most reads and feeds. It is fast, and a few seconds of staleness hurts nobody there. Sync is for the writes I cannot lose. Orders. Payments. Password changes. I do not pick one mode for the whole database. I pick per write, or per table, and I write down which one each table uses. The worst bug I have seen in this area was not a crash. It was a payments table running async because nobody had written the choice down, and a new engineer assumed it was sync.

Semi-sync sits in the middle. The primary waits for one replica, not all of them. I get most of the safety for a fixed cost. The catch is the timeout. If the replica does not confirm in time, the primary falls back to async and answers anyway. That fallback has to be loud. If it fails silently, I think I have sync and I do not.

Quorum writes are the stricter version. The primary waits until a majority of replicas confirm. That survives one dead replica without slowing every write to the slowest copy. It costs more in latency and in setup, and I only reach for it when losing a write and waiting both hurt.

Failure modes that actually happen

A replica that falls behind is the common one. A long report query blocks the replay, or a slow disk stalls it, and lag climbs while the primary looks fine. I alert on lag per replica, not on the average. One bad replica hides inside a good average every time.

A broken replication stream is quieter and worse. The primary and the replica both look healthy. They just disagree. I check the last shipped WAL position on the primary and the last applied position on each replica. If those two numbers drift apart, the stream is broken, even if every health check is green.

Split brain is the one I plan for and hope never to see. Two machines both think they are the primary, and both take writes. When they meet again, the writes conflict and somebody loses. The fix is fencing. The old primary must be stopped from taking writes before the new one starts. Promotion without fencing is not failover. It is a data loss plan with extra steps.

A promotion can also pick the wrong replica. The most caught up replica is the right one, not the nearest one and not the biggest one. I compare WAL positions before I promote, and I write down the gap. That gap is the list of writes I lost. I would rather tell the team that number in the first five minutes than discover it from users the next day.

What changes at 10x and 100x

At 10x the writes, replicas stop being the bottleneck for reads and start being a cost. Each replica replays every write, so ten replicas means ten replays of the same WAL. The primary spends real CPU shipping changes. I add replicas for read traffic, and I stop adding them when the replay cost on the primary starts to hurt write latency. That is usually sooner than people expect.

At 10x the reads, I also split the replicas by job. One replica serves the app reads. A different replica serves the reports. The report replica can lag a little and nobody cares, because reports are already looking at yesterday. The app replica must stay fresh. Mixing the two jobs on one replica is how a report takes down the user facing reads.

At 100x, replication alone does not save me. Replicas copy the whole database, so they do not fix a database that is too big or writes that are too fast. That is when I shard, which is Lesson 13. Replication and sharding solve different problems. Replication copies the data. Sharding splits it. I need both at real scale, and I need to know which problem I am solving before I add either one.

Cross region changes the numbers too. Inside one region, sync costs tens of milliseconds. Across regions it costs a hundred or more, because the speed of light does not negotiate. I keep sync inside one region and async across regions, and I say so in the runbook. A user in another region reading a replica there is reading the past by design, and the design has to own that.

# Primary: wait for one replica before answering a write.
# Async is the default if this line is missing, so check it.
synchronous_commit = on
synchronous_standby_names = 'replica_a'

# Replica: accept reads while it replays the log.
hot_standby = on

# Failover: fence the old primary first, then promote.
# Promote the replica with the highest applied WAL position.
# If the old primary can still take writes, it is not fenced yet.

A short replica and failover config, in the shape I actually keep in a runbook. The names are generic. The point is the three lines that matter: wait for one replica, name the replicas, and never promote without fencing.

I read that config on a bad day before I trust any dashboard. Dashboards show lag. The config shows what the ack actually promised.

One time it broke

A user updated their delivery address, reloaded the page, and saw the old address. They saved it again. Same old address. They opened a support ticket saying the site was eating their changes. The write was fine. The primary had the new address within milliseconds. The read after the write went to a replica that was 42 seconds behind, so the page showed the old address with a straight face. Nothing had failed. Nothing had errored. The user just could not see their own save.

Three things changed after, and I still do them now. First, reads right after a write go to the primary for a short window. A user must always see their own save, even if that costs the primary a few extra reads. Second, report queries moved to their own replica, so a report can never block the replica that serves users again. Third, lag alerts fire per replica at 2 seconds, not on the average across replicas. The average was fine that whole morning. One replica was 42 seconds behind, and the average hid it.

If you remember nothing else

Try designing it

Design replication for a small online store database. The store takes 500 writes per second at peak and 5,000 reads per second at peak. A lost order costs real money. A user must see their own order right after placing it. Other reads, like the product catalog, can be a few seconds stale and nobody will notice.

Clarifying questions to ask first

A worked approach

Trade-offs to name

Finished the lesson? Mark it complete and the course keeps your place on this device.

Next in this series
Lesson 13 — Sharding: Splitting the Data
Coming next