Platform Engineering · Lesson 39

Design a Notification System

Your phone buzzes. An email lands. A text arrives. Behind every one of those, one small event fanned out into a decision: who gets told, through which channel, and how many times. This lesson follows one notification from the event that triggered it to the receipt that proves it landed. It uses the queue from Lesson 15, the events from Lesson 16, and the event-driven flow from Lesson 28, and it shows where the whole thing breaks: duplicates, dead providers, and users you spammed without meaning to.

Difficulty: Advanced · Know this first: Lesson 15, Lesson 16, Lesson 28

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is the point. In the normal trip, one hop, the provider call, eats most of the budget. In the digest trip, the wait is the budget, and that is a choice, not an accident. Step through the lesson or switch the scenario and watch the bar change.

Try it: turn the knobs

Three knobs run this system. Move them and watch the delivery latency, the duplicate rate, and the provider cost move with them. The defaults below reproduce the normal scenario: one event, one push, delivered in about 260 ms.

Queue wait. The worker pool in this design drains about 10,000 notifications per second. Above that, events wait in the queue.

Fan-out. Each extra channel is another provider call for the same event, and another chance for a duplicate when dedupe keys differ per channel.

Worst-case retry cost. Each attempt waits with exponential backoff before the next try. This is the cost when the provider is failing, not when it is healthy.

Delivery latency
260 ms
Duplicate rate
0% of notifications delivered more than once
Provider cost
$0.10 per 1,000 notifications

The $0.10 per channel and $0.05 per retry are example blended rates for one provider call and one retry, not live prices. SMS costs far more than push in real life, so treat this as a teaching rate.

The notification system, from scratch: one event, one buzz

Extra diagram. I read the picture first: the boxes are the system, the moving dot is an event, and each step adds one piece until the full picture is there. No text needed to follow it.

Dedupe key: event id + user idDedupe store TTL 24 hChannels split into separate services at scale
12345678

Event pathChecksDeliveryMoving dots: request · response

The digest and the dead provider: two trips the diagram never shows

Extra diagram. I read the picture first: the boxes are the system, the moving dot is ten events, then one email; a message to a dead provider, and each step adds one piece until the full picture is there. No text needed to follow it.

Digest: 15 minutes or 50 eventsRetries capped at 3Dead-letter queue holds the rest
1234567

Digest pathRetry pathDead endMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The same topic, four times. One event, from the first buzz to the level where I can run 50 million users through dead providers and still not spam anyone.

Basic — One event, many possible messages

Definition: A notification system is a fan-out problem with a preference check in the middle. One small event fans out into a decision: who gets told, through which channel, and how many times. The preference check decides which of the possible messages are real. Quiet is a feature, and the user controls the volume.

Types:

  • Push, fast and cheap, to the phone.
  • Email, slower, for things that can wait.
  • SMS, which reaches everyone and costs the most.
  • Digest, where events accumulate for a window and go out as one message instead of many.

Uses: I use this level for the first version of a notification service. One event lands in a queue, the service reads it, checks preferences, picks a channel, and sends. The normal trip in this lesson is one event, one push, delivered in about 260 ms.

Dimensions: As load grows, the worker pool is the limit: in this lesson it drains about 10,000 notifications a second, and above that, events wait in the queue. What breaks is duplicates, one event delivered twice because the dedupe check missed. What it costs is provider spend, and SMS is the channel where a bug spends real money. What I give up is speed when I batch: digest is slower by design, and urgent events must bypass it.

Intermediate — Dedupe is a promise

Definition: Dedupe is a product decision, not a bug fix. One event, one message, enforced by a key built from the event id and the user id, never from the send attempt, stored with a 24-hour time to live. Delivery receipts are the only proof a message landed. Sent is not delivered.

Types:

  • A dedupe store, keyed by event and user, checked on the hot path of every send.
  • Retries capped at three, with backoff of 1 second, then 2, then 4, and then the event goes to the dead-letter queue.
  • A dead-letter queue, where a message I can still deliver waits instead of being dropped silently.
  • Delivery receipts, stored for every send, so I can prove what landed.

Uses: I use this level in the incident in this lesson. A deploy changed how the dedupe key was built. The old key used the event id and the user id. The new key added the channel name to it. That one-line change meant the same event now had a different key on each channel. The dedupe check never matched. Every user who had push and email enabled got every alert twice. Users with SMS enabled got it three times. The alerts went out at 3 AM during a maintenance window, so the first reports came from users who were woken up, not from our dashboards.

Dimensions: As load grows, the dedupe lookup is a hot-path read, and a slow one adds latency to every send. What breaks is a change to the key format: a one-line change breaks the promise to the user. What it costs is the dedupe cluster and the receipt storage. What I give up is exactly-once delivery, which the queue cannot promise; the queue guarantees at-least-once, so duplicates are possible and the dedupe key is what makes them rare.

Advanced — Channels split apart

Definition: At a hundred times the scale, the channels split into separate services with separate queues, so a slow email provider cannot block urgent push alerts. That is a bulkhead, from Lesson 23, applied to notifications. The preference check becomes its own read model, cached and refreshed, because it runs on every single event and the user database cannot take that read load.

Types:

  • Per-channel queues and workers, one for push, one for email, one for SMS.
  • A cached preference read model, refreshed on its own schedule.
  • Push token cleanup as a background job, not something the send path does, because tokens rotate and die.
  • An SMS daily budget guard, because one bad routing rule can spend thousands of dollars before anyone notices.

Uses: I use this level for the design exercise in this lesson, 50 million users across push, email, and SMS. The sizing first: 50 million users at 5 notifications each per day is 250 million a day, about 2,900 a second on average, and a ten-times peak is about 29,000 a second, so the queue and the worker pool must drain 30,000 a second with headroom. Digest runs as a scheduled batch: events for digest users accumulate for 15 minutes or up to 50 events, then one email goes out.

Dimensions: As load grows, the dedupe store becomes its own cluster, because it is a hot-path read on every send. What breaks is a provider going down: aggressive retries recover more messages and also hurt a sick provider more, so I cap retries at three and let the dead-letter queue hold the rest. What it costs is a service per channel and the budget guard. What I give up is perfect ordering; users expect the digest to read in time order, but events arrive out of order across regions, so I keep a short holdback window of a few seconds and accept that perfect order costs latency I do not want to pay.

Master — Running notifications when the provider fails

Definition: At this level the dedupe key format is versioned, and a change to it is a reviewed change, not a drive-by edit. There is an alert on the duplicate rate itself: if the same event id is delivered twice within an hour, the on-call gets paged. Notification deploys go out behind a flag with a canary, like any other user-facing change. Dedupe is a promise to the user, and a change to the promise needs the same care as a change to the product.

Types:

  • Versioned dedupe key formats, with changes reviewed like product changes.
  • A duplicate-rate alert, paging when the same event id is delivered twice in an hour.
  • Flagged, canaried notification deploys, so a bad key format reaches a few users first.
  • Digest windows per user: power users get hourly, quiet users get daily, and the window is a preference, not a constant.

Uses: I use this level after the 3 AM incident in this lesson. The rollback shipped the old key format and duplicates stopped within minutes, but the review the next morning is what mattered: the duplicate rate for the night was calculated and shared with the team, and the three changes above landed because of it. The three metrics on the on-call dashboard are the duplicate rate, the dead-letter depth, and the SMS spend.

Dimensions: As load grows, the preference check runs on every single event, so it is the first thing to get its own read model. What breaks is ordering across regions: events arrive out of order, and the holdback window is the only fix that does not cost a redesign. What it costs is the versioning discipline, the alerting, and the canary on every notification deploy. What I give up is the quick deploy; a notification change is a user-facing change, and it ships like one.

Go deeper

At small scale, one notification service and one queue carry the whole system. At 10x, the pieces start to split. The preference check becomes its own read model, cached and refreshed, because it runs on every single event and the user database cannot take that read load. The dedupe store becomes its own cluster, because a dedupe lookup is a hot-path read and a slow one adds latency to every send. At 100x, the channels split into separate services with separate queues, so a slow email provider cannot block urgent push alerts. That is a bulkhead, from Lesson 23, applied to notifications. Push tokens rotate and die, so token cleanup is a background job, not something the send path does. SMS gets a budget guard, because one bad routing rule can spend thousands of dollars before anyone notices. And the digest is not one window. Power users get hourly, quiet users get daily, and the window is a preference, not a constant. The hardest trade-off at scale is ordering. Users expect the digest to read in time order, but events arrive out of order across regions. I keep a short holdback window, a few seconds, to let late events land, and I accept that perfect order costs latency I do not want to pay. The other trade-off is the retry policy. Aggressive retries recover more messages and also hurt a sick provider more. I cap retries at three, back off exponentially, and let the dead-letter queue hold the rest. A message in the dead-letter queue is a message I can still deliver. A message dropped silently is gone.

Notification service policy

notification:
  dedupe:
    key: "evt:{event_id}:user:{user_id}"
    ttl_seconds: 86400
  retry:
    max_attempts: 3
    backoff_seconds: [1, 2, 4]
    dead_letter_after: 3
  digest:
    window_minutes: 15
    max_events_per_digest: 50
  channels:
    push: { provider: fcm, timeout_ms: 2000 }
    email: { provider: ses, timeout_ms: 5000 }
    sms: { provider: twilio, timeout_ms: 5000, daily_budget_usd: 500 }

The dedupe key is built from the event id and the user id, never from the send attempt. The retry policy caps at three attempts with 1s, 2s, and 4s of backoff, then the event goes to the dead-letter queue. The SMS channel carries a daily budget, because SMS is the one channel where a bug spends real money.

One time it broke

A deploy changed how the dedupe key was built. The old key used the event id and the user id. The new key added the channel name to it. That one-line change meant the same event now had a different key on each channel. The dedupe check never matched. Every user who had push and email enabled got every alert twice. Users with SMS enabled got it three times. The alerts went out at 3 AM during a maintenance window, so the first reports came from users who were woken up, not from our dashboards.

Timeline

  • 03:12Deploy ships the new dedupe key format. No flag, no canary, full rollout.
  • 03:14First duplicate alerts go out. Push and email for the same event, minutes apart.
  • 03:40Support tickets start arriving. Users report double alerts. On-call checks the provider dashboard first and sees nothing wrong.
  • 04:05On-call compares dedupe keys from two copies of the same alert and sees they differ by channel. Root cause found.
  • 04:20Rollback ships the old key format. Duplicates stop.
  • 09:00Morning review. The duplicate rate for the night is calculated and shared with the team.

What changed after

Three things. First, the dedupe key format is versioned, and a change to it is a reviewed change, not a drive-by edit. Second, we added an alert on the duplicate rate itself: if the same event id is delivered twice within an hour, the on-call gets paged. Third, notification deploys go out behind a flag with a canary, like any other user-facing change. The lesson I kept is simple. Dedupe is a promise to the user. A change to the promise needs the same care as a change to the product.

If you remember nothing else

Design exercise

Design a notification system for 50 million users across push, email, and SMS, with dedupe and digest. Sketch the flow from event to delivery, and name the parts that break first at scale.

Clarifying questions to ask

  • How many notifications per user per day, and which kinds are urgent?
  • What does "delivered" mean for each channel, and how do we prove it?
  • Which channels can a user mute, and do quiet hours apply to all of them or only some?
  • How long can a digest wait, and who picks the window?
  • What is the SMS budget, and who gets paged when it is close?

Trade-offs to name

  • Latency against quiet: digest is slower by design, and urgent events must bypass it.
  • Cost against reach: SMS reaches everyone and costs the most, so it is reserved for things the user asked for by name.
  • At-least-once against exactly-once: the queue guarantees at-least-once delivery, so duplicates are possible and the dedupe key is what makes them rare.
  • Ordering against speed: a short holdback window buys in-order digests and costs a few seconds.

Worked approach

Start with the event, not the message. Events land in a queue, the queue from Lesson 15, and a notification service reads them, the event-driven flow from Lesson 28. Sizing first: 50 million users at 5 notifications each per day is 250 million a day, about 2,900 per second on average, and a 10x peak is about 29,000 per second, so the queue and the worker pool must drain 30,000 per second with headroom. The service checks preferences, picks a channel, and dedupes on a key built from event id and user id, stored with a 24-hour TTL. Push, email, and SMS get separate queues and separate workers, so a slow email provider cannot block urgent push. That is the bulkhead from Lesson 23. Digest runs as a scheduled batch: events for digest users accumulate for 15 minutes or up to 50 events, then one email goes out. Retries cap at three with 1s, 2s, and 4s of backoff, then the event goes to the dead-letter queue for replay. Receipts are stored for every send, and the duplicate rate, the dead-letter depth, and the SMS spend are the three metrics on the on-call dashboard.

Saved in this browser only, so the Lessons index can show your progress across the course.

Next in this series
Lesson 40 — Design a Ride-Hailing System