A customer places an order. One thing happens, and five different parts of the system need to know about it. The payment service needs to charge the card. The inventory service needs to hold the item. The shipping service needs to print a label. The notification service needs to send the confirmation. The analytics service needs to count the sale. In Lesson 15 the answer was a queue, and a queue gives one job to one worker. That does not work here. One event has to reach every subscriber, and none of them should have to know the others exist. That is pub/sub. This lesson follows one published event from the publisher, through the broker and the topic, fanning out to four subscribers, and shows what happens when one subscriber is slow, and what happens when a subscriber crashes and has to replay.
Difficulty: Advanced · Know this first: Lesson 01, Lesson 15
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in a healthy trip the subscribers do the real work and the broker's hops are nearly free. Switch the scenario and watch the bar change. In the slow subscriber scenario, one hop eats almost the whole budget, and it eats it inside one group while the other groups stay fast. That split is the lesson. There is no single total for a pub/sub system. There is one total per group.
Extra diagram · from scratch to master
Start with one publisher and one subscriber and no topic at all. Add the topic as a log, then the partitions, then the four subscriber groups.
Extra diagram · from scratch to master
Start from the calm groups, each reading from its own offset. Then the notification subscriber restarts from the earliest offset and replays a week of old events.
The budget above is one fixed trip. This box is yours. Move the sliders and watch what happens to one event when the publish rate climbs, when more subscribers join, and when one subscriber gets slow. I learned this the slow way: fan-out is free to add and expensive to run, and the slowest subscriber in a group decides what the group feels.
How fast events land in the log, so how hard the append hop works, and how fast the slow subscriber's partition fills up. The topic here has 32 partitions.
How many copies the broker has to deliver for every event, so how long the fan-out hop takes.
The time the slow subscriber takes on each event. This is the Step 8 timing directly, and it decides how far behind its group falls.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above follows one published event from the publisher, through the broker and the topic, fanning out to four subscribers. This section covers the same fan-out four times, from the first subscription to the level where I run many topics and rehearse the replay.
Definition: One thing happens, and five different parts of the system need to know about it. The payment service charges the card. The inventory service holds the item. The shipping service prints a label. The notification service sends the confirmation. The analytics service counts the sale. A queue gives one job to one worker. That does not work here. One event has to reach every subscriber, and none of them should have to know the others exist. That is pub/sub.
Types:
Uses: I use this level when several parts of the system each need to react to the same fact. I pick the queue when the question is who does the work. I pick the topic when the question is who needs to know.
Dimensions:
Definition: The log keeps every event until retention expires, and the offset is each group's bookmark. That bookmark is what makes replay possible. A subscriber that crashes comes back, reads from its last committed offset, and replays what it missed. The rule is simple. Read, do the work, then commit the offset. Never commit before the work is done.
Types:
Uses: I use this level when a subscriber crashes and comes back. I check its committed offset first. If the offset is behind, it replays. If the subscriber has no idempotency check, the replay sends every message twice, and the users get the duplicates.
Dimensions:
Definition: Ordering holds inside a partition. It does not hold across partitions, and it does not hold across subscriber groups, because each group reads at its own speed. It also breaks on replay. A subscriber that replays old events sees them in order, but it sees them late. A slow subscriber is the quietest failure here. It does not error. It does not crash. It just takes longer on each event, and the lag grows so slowly that nobody notices for an hour.
Types:
Uses: I use this level when two events for the same order have to arrive in order. I put all of that order's events on one key, in one partition, and I accept that one partition is the bottleneck for that order. If the business needs order across everything, I have the wrong tool, and I say so early.
Dimensions:
Definition: At a hundred times the events, one topic is not one problem any more. I split topics by what changes together. The order topic stays small and fast. The analytics events go to their own topic, where a slow consumer cannot touch the order flow. Schema changes get a real process. New fields are optional, old fields are never renamed in place, and every version is tested against every subscriber before it ships. And replay becomes a planned operation, not a panic button. I rehearse it, because the first time I replay a hundred times the events should not be during an incident.
Types:
Uses: I use this level when one topic's traffic starts hurting another's. I split the topics, I put the schema process in writing, and I rehearse the replay. The lesson's incident is why. A deploy reset one group's offset, and the fix was an idempotency check, a second pair of eyes on offset changes, and an alert on offset jumps.
Dimensions:
The lesson above is one event and four subscribers on a good day. Production is the rest of it. Here is what I actually spend my time on once the topic is running.
A queue gives one job to one worker. I use it when the work should be done once, by whoever is free. Pub/sub gives one event to every subscriber. I use it when several parts of the system each need to react to the same fact. The mistake I see most is using pub/sub for work that should be done once. Then every subscriber does the job, or they fight over who does it, and the duplicates look like a bug. They are not a bug. They are the design I chose. I pick the queue when the question is who does the work. I pick the topic when the question is who needs to know.
Ordering holds inside a partition. It does not hold across partitions, and it does not hold across subscriber groups, because each group reads at its own speed. It also breaks on replay. A subscriber that replays old events sees them in order, but it sees them late, after newer events may already have been processed by another group. If the business needs strict order for one order, I put all of that order's events on one key, in one partition, and I accept that one partition is the bottleneck for that order. If the business needs order across everything, I have the wrong tool, and I say so early.
A slow subscriber is the first one. It does not error. It does not crash. It just takes longer on each event, and the lag grows so slowly that nobody notices for an hour. The second is a poison event. One event that makes a subscriber crash every time it reads it. The subscriber restarts, reads the same event, crashes again. The group is stuck on one offset forever. The fix is a dead-letter queue: after a few tries, the event moves aside and the group moves on, and a human looks at the poison event later. The third is an offset reset after a deploy. Somebody sets the group to read from the earliest offset, and the next morning every subscriber replays a week of events. Users get a week of duplicate messages. I treat offset changes like a production deploy, with a second pair of eyes.
The log keeps events until retention expires. Longer retention means more disk and more money, and it also means a crashed subscriber can replay further back. Short retention means cheap disk and a subscriber that crashes on a Friday and comes back Monday to find its events already gone. I size retention by the worst recovery I am willing to pay for: how far back would I ever need to replay, and what does that disk cost.
At ten times the events, I add partitions before I add brokers, because partitions are what let more subscribers read in parallel. I watch lag per group as a first-class metric, with an alert, not as a number I check when someone complains. And I make every subscriber idempotent before the traffic grows, because duplicates at ten times the volume are ten times the angry users.
At a hundred times the events, one topic is not one problem any more. I split topics by what changes together. The order topic stays small and fast. The analytics events go to their own topic, where a slow consumer cannot touch the order flow. Schema changes get a real process: new fields are optional, old fields are never renamed in place, and every version is tested against every subscriber before it ships. And replay becomes a planned operation, not a panic button. I rehearse it, because the first time I replay a hundred times the events should not be during an incident.
# Subscriber: read, do the work, THEN commit the offset. # Never commit before the work is done. auto_commit: false commit_after: process max_retries: 3 on_exhausted_retries: send_to_dead_letter # Dead-letter topic: orders.dlq # A subscriber that crashes on the same event 3 times # stops retrying it and moves on. A human reads the DLQ. # Idempotency: check the event id before doing the work. # If this event id was already processed, skip it and commit.
Committing after the work is the whole trick. Commit before the work and a crash loses the event. Never commit at all and every restart replays everything. Commit after the work and a crash costs me one duplicate, which the idempotency check absorbs.
This one stayed with me because the system did everything it was designed to do, and users still got hurt. The design was right. The deploy was wrong.
A deploy to the notification subscriber changed how it read its offset. The new version started the group from the earliest offset instead of the last committed one. The topic kept a week of events. So the subscriber woke up, read a week of old order events, and started sending confirmation messages for orders that were already delivered days ago. Thousands of users got a message about an order they had already received. Nothing crashed. No error rate moved. The broker was healthy, the publisher was healthy, and the lag metric even looked good, because the subscriber was chewing through the backlog fast. The first sign of trouble was a user writing in to ask why they got a shipping message for a package already on their doorstep.
Three things, and I still do all three. First, every subscriber gets an idempotency check before it does any real work. A duplicate event should be a no-op, not a second message to a user. Second, offset changes are treated like production deploys. Nobody changes where a group starts reading without a second person reviewing it, and the change is tested on a quiet topic first. Third, I added an alert on offset jumps, not just on lag. Lag going up is slow and visible. An offset jumping backwards by a week is sudden and silent, and it is the one I want to be paged for.
This is the kind of question I would ask in an interview, and the kind I ask myself before I add a new subscriber to a real topic. Take a minute with it before you read my approach.
The prompt. You run an online store. When a customer places an order, five services need to know: payment, inventory, shipping, notifications, and analytics. Peak is 5,000 orders a minute. Each order produces one order.placed event. Design the pub/sub for this. How many topics and partitions do you use, which subscribers share a group and which get their own, and what do you do when the analytics subscriber falls behind?
I would ask four things before I sized anything. Does any subscriber need the events in strict order, or is rough order fine? How long can each subscriber afford to be behind: seconds for payment, minutes for analytics? What happens if an event is delivered twice: can payment charge the card twice, or is it safe? And how far back would I ever need to replay: hours, or a full week?
5,000 orders a minute is about 83 events a second. That is small for a broker, so the topic is not the bottleneck. The subscribers are. I use one topic, orders, with 16 partitions, keyed by order id, so all the events for one order stay in order in one partition. Payment, inventory, shipping, and notifications each get their own subscriber group, because each one has to see every event and each one moves at its own speed. Analytics gets its own group too, and its own topic is not needed yet at this volume, but I keep it separate in my head: if analytics falls behind, it must never slow payment, and separate groups guarantee that. 83 events a second across 16 partitions is about 5 events a second per partition, so even a slow subscriber at 25 ms per event keeps up easily. The numbers only get interesting when one subscriber gets slow or one group replays.
I keep payment in its own group and I make it idempotent first, because a duplicate charge is the one mistake I cannot apologize away. I let analytics fall behind on purpose. It reads the same events, but nobody's checkout waits on it, so a slow analytics subscriber is a stale dashboard, not an incident. I set retention to seven days, so a crashed subscriber can replay a full week, and I pay the disk cost for that because the alternative is lost events. I add a dead-letter queue for the poison event, so one bad order cannot wedge a whole group. And I alert on lag per group, with a tighter threshold for payment than for analytics, because not all lag is equal.
Saved on this device, for this lesson only.