Platform Engineering · Lesson 16

Shout Once, Heard by Many

A customer places an order. One thing happens, and five different parts of the system need to know about it. The payment service needs to charge the card. The inventory service needs to hold the item. The shipping service needs to print a label. The notification service needs to send the confirmation. The analytics service needs to count the sale. In Lesson 15 the answer was a queue, and a queue gives one job to one worker. That does not work here. One event has to reach every subscriber, and none of them should have to know the others exist. That is pub/sub. This lesson follows one published event from the publisher, through the broker and the topic, fanning out to four subscribers, and shows what happens when one subscriber is slow, and what happens when a subscriber crashes and has to replay.

Difficulty: Advanced · Know this first: Lesson 01, Lesson 15

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in a healthy trip the subscribers do the real work and the broker's hops are nearly free. Switch the scenario and watch the bar change. In the slow subscriber scenario, one hop eats almost the whole budget, and it eats it inside one group while the other groups stay fast. That split is the lesson. There is no single total for a pub/sub system. There is one total per group.

Extra diagram · from scratch to master

One topic, four groups: the log and its readers

Start with one publisher and one subscriber and no topic at all. Add the topic as a log, then the partitions, then the four subscriber groups.

topic: 32 partitions4 subscriber groupsoffsets 1040, 1041, 1042
Order service
the publisher
The topic
a log, not a queue
Partitions
32, by hashing the order id
Broker
leader plus replicas
Notifications
its own offset
Shipping
its own offset
Search
slow, its partition fills
Analytics
its own offset
Request / flowBuild / fixWaiting / queueResponse · the way backFailure / dead end

Extra diagram · from scratch to master

Replay gone wrong: the week of old confirmations

Start from the calm groups, each reading from its own offset. Then the notification subscriber restarts from the earliest offset and replays a week of old events.

replay: a week of old ordersconfirmations sent for days-old ordersfix: restart from the committed offset
Normal delivery
each group reads its offset
Subscriber crashes
notifications go down
Restarts wrong
from the earliest offset
A week replays
old confirmations sent
The fix
restart from the committed offset
Slow search
its partition fills
Request / flowFailure / dead endBuild / fixWaiting / queueResponse · the way back

Move the load and watch the slow subscriber fall behind

The budget above is one fixed trip. This box is yours. Move the sliders and watch what happens to one event when the publish rate climbs, when more subscribers join, and when one subscriber gets slow. I learned this the slow way: fan-out is free to add and expensive to run, and the slowest subscriber in a group decides what the group feels.

How fast events land in the log, so how hard the append hop works, and how fast the slow subscriber's partition fills up. The topic here has 32 partitions.

How many copies the broker has to deliver for every event, so how long the fan-out hop takes.

The time the slow subscriber takes on each event. This is the Step 8 timing directly, and it decides how far behind its group falls.

One event, end to end
~82 ms
Fan-out deliveries / sec
4,000
Per-partition rate
~31 / sec
Slow subscriber ceiling
40 / sec

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The lesson above follows one published event from the publisher, through the broker and the topic, fanning out to four subscribers. This section covers the same fan-out four times, from the first subscription to the level where I run many topics and rehearse the replay.

Basic — One event, every subscriber

Definition: One thing happens, and five different parts of the system need to know about it. The payment service charges the card. The inventory service holds the item. The shipping service prints a label. The notification service sends the confirmation. The analytics service counts the sale. A queue gives one job to one worker. That does not work here. One event has to reach every subscriber, and none of them should have to know the others exist. That is pub/sub.

Types:

  • The publisher — the service that publishes the event and moves on.
  • The topic — the named log the event is appended to.
  • The subscriber group — the set of services that each get their own copy.
  • The broker — the middle that stores the event and fans it out.

Uses: I use this level when several parts of the system each need to react to the same fact. I pick the queue when the question is who does the work. I pick the topic when the question is who needs to know.

Dimensions:

  • Load: At 1,000 events a second across 4 subscriber groups, the broker delivers 4,000 copies a second. The topic has 32 partitions, so each partition carries about 31 events a second. Fan-out is free to add and expensive to run.
  • Breaks: It breaks when I use pub/sub for work that should be done once. Then every subscriber does the job, or they fight over who does it, and the duplicates look like a bug. They are not a bug. They are the design I chose.
  • Cost: The cost is one delivery per group per event. Every group I add multiplies the broker's work. Four groups at 1,000 events a second is 4,000 deliveries a second.
  • Trade-off: I give up one worker doing the job once to get every subscriber hearing the same fact. The price is copies, and a slow subscriber that nobody has to wait for yet still has to be watched.

Intermediate — Offsets, replay, and commit after the work

Definition: The log keeps every event until retention expires, and the offset is each group's bookmark. That bookmark is what makes replay possible. A subscriber that crashes comes back, reads from its last committed offset, and replays what it missed. The rule is simple. Read, do the work, then commit the offset. Never commit before the work is done.

Types:

  • The offset — the bookmark that says where a group stopped reading.
  • The commit — the write that moves the bookmark, done after the work.
  • The replay — the re-read of events after a crash, from the last committed offset.
  • The idempotency check — the event id checked before the work, so a replayed event is a no-op.

Uses: I use this level when a subscriber crashes and comes back. I check its committed offset first. If the offset is behind, it replays. If the subscriber has no idempotency check, the replay sends every message twice, and the users get the duplicates.

Dimensions:

  • Load: At 10x the events I make every subscriber idempotent before the traffic grows, because duplicates at ten times the volume are ten times the angry users.
  • Breaks: It breaks when a deploy resets a group to the earliest offset. The subscriber wakes up, reads a week of old events, and sends confirmations for orders already delivered days ago. I treat offset changes like a production deploy, with a second pair of eyes. I notice it when the committed offset jumps backwards by a week.
  • Cost: The cost is the retention disk and the idempotency store per subscriber. Longer retention means more disk and more money, and it also means a crashed subscriber can replay further back.
  • Trade-off: I give up exactly-once, because the broker delivers at least once, never exactly once, to get replay I can trust. The price is a commit order I never get wrong and a check that makes every duplicate harmless.

Advanced — Ordering, slow subscribers, and poison events

Definition: Ordering holds inside a partition. It does not hold across partitions, and it does not hold across subscriber groups, because each group reads at its own speed. It also breaks on replay. A subscriber that replays old events sees them in order, but it sees them late. A slow subscriber is the quietest failure here. It does not error. It does not crash. It just takes longer on each event, and the lag grows so slowly that nobody notices for an hour.

Types:

  • The partition key — the field that puts all of one order's events on one partition, in order.
  • The slow subscriber — the one that takes longer per event, so its group falls behind while the others stay fast.
  • The poison event — the one event that makes a subscriber crash every time it reads it.
  • Per-group lag — the metric I watch, because a total hides the one group that is behind.

Uses: I use this level when two events for the same order have to arrive in order. I put all of that order's events on one key, in one partition, and I accept that one partition is the bottleneck for that order. If the business needs order across everything, I have the wrong tool, and I say so early.

Dimensions:

  • Load: At ten times the events I add partitions before I add brokers, because partitions are what let more subscribers read in parallel. The slow subscriber's ceiling is what it is. At 25 ms per event it handles 40 a second, and no broker setting changes that.
  • Breaks: It breaks when a poison event sticks a group on one offset forever. The subscriber restarts, reads the same event, crashes again. After a few tries the event moves to the dead-letter queue and the group moves on, and a human looks at it later.
  • Cost: The cost is lag monitoring per group and the dead-letter topic per group. A single lag number for the whole system is cheap and useless.
  • Trade-off: I give up global order to get parallel reads. The price is ordering that only holds inside one partition, and a slow subscriber that sets the pace for its whole group.

Master — Many topics, schema rules, and rehearsed replays

Definition: At a hundred times the events, one topic is not one problem any more. I split topics by what changes together. The order topic stays small and fast. The analytics events go to their own topic, where a slow consumer cannot touch the order flow. Schema changes get a real process. New fields are optional, old fields are never renamed in place, and every version is tested against every subscriber before it ships. And replay becomes a planned operation, not a panic button. I rehearse it, because the first time I replay a hundred times the events should not be during an incident.

Types:

  • The split topic — order events on one topic, analytics on another, so a slow consumer cannot touch the order flow.
  • The schema rule — new fields optional, old fields never renamed, every version tested against every subscriber.
  • The offset-jump alert — the alarm that fires when a group's offset jumps backwards, not just when lag climbs.
  • The rehearsed replay — the replay I have run before, on a quiet topic, before I ever need it in an incident.

Uses: I use this level when one topic's traffic starts hurting another's. I split the topics, I put the schema process in writing, and I rehearse the replay. The lesson's incident is why. A deploy reset one group's offset, and the fix was an idempotency check, a second pair of eyes on offset changes, and an alert on offset jumps.

Dimensions:

  • Load: At 100x the events the order topic stays small and fast because the analytics events live elsewhere. Each topic scales on its own terms, and a slow analytics consumer cannot slow the order flow.
  • Breaks: It breaks when a schema change ships untested and a subscriber cannot read the new events. I notice it when one group's lag climbs and the others stay flat, and the cause is a version the subscriber never saw in testing.
  • Cost: The cost is topics to run, schema versions to test, and replays to rehearse. Every split is a new topic to monitor and a new retention bill.
  • Trade-off: I give up one topic I can hold in my head to get failures that stay in their own topic. The price is a schema process and a replay runbook, and both need a human to keep them honest.

Go deeper

The lesson above is one event and four subscribers on a good day. Production is the rest of it. Here is what I actually spend my time on once the topic is running.

Topics versus queues, and when I pick which

A queue gives one job to one worker. I use it when the work should be done once, by whoever is free. Pub/sub gives one event to every subscriber. I use it when several parts of the system each need to react to the same fact. The mistake I see most is using pub/sub for work that should be done once. Then every subscriber does the job, or they fight over who does it, and the duplicates look like a bug. They are not a bug. They are the design I chose. I pick the queue when the question is who does the work. I pick the topic when the question is who needs to know.

Ordering is a promise with edges

Ordering holds inside a partition. It does not hold across partitions, and it does not hold across subscriber groups, because each group reads at its own speed. It also breaks on replay. A subscriber that replays old events sees them in order, but it sees them late, after newer events may already have been processed by another group. If the business needs strict order for one order, I put all of that order's events on one key, in one partition, and I accept that one partition is the bottleneck for that order. If the business needs order across everything, I have the wrong tool, and I say so early.

Failure modes that do not look like failures

A slow subscriber is the first one. It does not error. It does not crash. It just takes longer on each event, and the lag grows so slowly that nobody notices for an hour. The second is a poison event. One event that makes a subscriber crash every time it reads it. The subscriber restarts, reads the same event, crashes again. The group is stuck on one offset forever. The fix is a dead-letter queue: after a few tries, the event moves aside and the group moves on, and a human looks at the poison event later. The third is an offset reset after a deploy. Somebody sets the group to read from the earliest offset, and the next morning every subscriber replays a week of events. Users get a week of duplicate messages. I treat offset changes like a production deploy, with a second pair of eyes.

Retention is a bill and a safety net

The log keeps events until retention expires. Longer retention means more disk and more money, and it also means a crashed subscriber can replay further back. Short retention means cheap disk and a subscriber that crashes on a Friday and comes back Monday to find its events already gone. I size retention by the worst recovery I am willing to pay for: how far back would I ever need to replay, and what does that disk cost.

What changes at 10x

At ten times the events, I add partitions before I add brokers, because partitions are what let more subscribers read in parallel. I watch lag per group as a first-class metric, with an alert, not as a number I check when someone complains. And I make every subscriber idempotent before the traffic grows, because duplicates at ten times the volume are ten times the angry users.

What changes at 100x

At a hundred times the events, one topic is not one problem any more. I split topics by what changes together. The order topic stays small and fast. The analytics events go to their own topic, where a slow consumer cannot touch the order flow. Schema changes get a real process: new fields are optional, old fields are never renamed in place, and every version is tested against every subscriber before it ships. And replay becomes a planned operation, not a panic button. I rehearse it, because the first time I replay a hundred times the events should not be during an incident.

Subscriber config I actually use

# Subscriber: read, do the work, THEN commit the offset.
# Never commit before the work is done.
auto_commit: false
commit_after: process
max_retries: 3
on_exhausted_retries: send_to_dead_letter
# Dead-letter topic: orders.dlq
# A subscriber that crashes on the same event 3 times
# stops retrying it and moves on. A human reads the DLQ.
# Idempotency: check the event id before doing the work.
# If this event id was already processed, skip it and commit.

Committing after the work is the whole trick. Commit before the work and a crash loses the event. Never commit at all and every restart replays everything. Commit after the work and a crash costs me one duplicate, which the idempotency check absorbs.

One time it broke

This one stayed with me because the system did everything it was designed to do, and users still got hurt. The design was right. The deploy was wrong.

What broke

A deploy to the notification subscriber changed how it read its offset. The new version started the group from the earliest offset instead of the last committed one. The topic kept a week of events. So the subscriber woke up, read a week of old order events, and started sending confirmation messages for orders that were already delivered days ago. Thousands of users got a message about an order they had already received. Nothing crashed. No error rate moved. The broker was healthy, the publisher was healthy, and the lag metric even looked good, because the subscriber was chewing through the backlog fast. The first sign of trouble was a user writing in to ask why they got a shipping message for a package already on their doorstep.

Timeline

What changed after

Three things, and I still do all three. First, every subscriber gets an idempotency check before it does any real work. A duplicate event should be a no-op, not a second message to a user. Second, offset changes are treated like production deploys. Nobody changes where a group starts reading without a second person reviewing it, and the change is tested on a quiet topic first. Third, I added an alert on offset jumps, not just on lag. Lag going up is slow and visible. An offset jumping backwards by a week is sudden and silent, and it is the one I want to be paged for.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I add a new subscriber to a real topic. Take a minute with it before you read my approach.

The prompt. You run an online store. When a customer places an order, five services need to know: payment, inventory, shipping, notifications, and analytics. Peak is 5,000 orders a minute. Each order produces one order.placed event. Design the pub/sub for this. How many topics and partitions do you use, which subscribers share a group and which get their own, and what do you do when the analytics subscriber falls behind?

Clarifying questions to ask first

I would ask four things before I sized anything. Does any subscriber need the events in strict order, or is rough order fine? How long can each subscriber afford to be behind: seconds for payment, minutes for analytics? What happens if an event is delivered twice: can payment charge the card twice, or is it safe? And how far back would I ever need to replay: hours, or a full week?

Rough sizing, worked

5,000 orders a minute is about 83 events a second. That is small for a broker, so the topic is not the bottleneck. The subscribers are. I use one topic, orders, with 16 partitions, keyed by order id, so all the events for one order stay in order in one partition. Payment, inventory, shipping, and notifications each get their own subscriber group, because each one has to see every event and each one moves at its own speed. Analytics gets its own group too, and its own topic is not needed yet at this volume, but I keep it separate in my head: if analytics falls behind, it must never slow payment, and separate groups guarantee that. 83 events a second across 16 partitions is about 5 events a second per partition, so even a slow subscriber at 25 ms per event keeps up easily. The numbers only get interesting when one subscriber gets slow or one group replays.

Trade-offs to name

I keep payment in its own group and I make it idempotent first, because a duplicate charge is the one mistake I cannot apologize away. I let analytics fall behind on purpose. It reads the same events, but nobody's checkout waits on it, so a slow analytics subscriber is a stale dashboard, not an incident. I set retention to seven days, so a crashed subscriber can replay a full week, and I pay the disk cost for that because the alternative is lost events. I add a dead-letter queue for the poison event, so one bad order cannot wedge a whole group. And I alert on lag per group, with a tighter threshold for payment than for analytics, because not all lag is equal.

Saved on this device, for this lesson only.

Next in this series
Lesson 17 — Live Data, As It Happens