Platform Engineering · Lesson 27

The Big Map, Then the Small Details

Every design interview I have sat in, and every real system I have built, splits into two questions. The first is the big picture: what are we building, how big is it, what breaks first. The second is the small picture: how does this one piece actually work, line by line, edge case by edge case. The first is the high-level design. The second is the low-level design. I have watched strong engineers fail interviews by jumping to the second question before the first one had an answer. This lesson walks one small problem, a notification service, through both. HLD first, then LLD, in that order, and I will show you what each one has to prove before you move on.

Difficulty: Advanced · Know this first: Lesson 02, Lesson 03

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

The HLD, as boxes

One trigger, one sender, one delivery. I add the queue and the database, then the four workers and the fallback.

Workers: 4, with leasesDedupe key: message id + recipient idPush first: email as fallback
dedupe checkif push failsTriggerorder ships, paymentfails, delivery lateAPItakes the event inQueueevents wait hereSender workers x4with leaseDeliveries databasededupe key: messageid + recipient idPush providerfirstEmail providerfallback
the notification event
notification eventdeliveredwaiting in queueduplicate blockedrequest · flowingresponse · the way back

    Extra diagram · Second trip

    Designing it: HLD first, then LLD

    The design itself is the trip here. I follow the requirements as they become boxes, then a contract, then checked edge cases.

    Start: one questionMath: messages, QPS, storageInvariant: delivered never moves back
    Requirements listedwhat are we buildingNapkin mathmessages, QPS,storageBoxes drawnAPI, queue, sender,databaseData modeldelivery row, dedupekeyFailure listwritten downLLD contractsender function,invariantsEdge cases checkedthe 10x scenario
    the design
    the designcontract donestill a questionrequest · flowingresponse · the way back

      Where the time goes

      Each hop owns a slice of the end-to-end latency budget for the notification service. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: fan-out eats most of the budget, because one message becomes many deliveries. The trigger and the queue take no slice of the bar. The trigger is an event, not work, and putting a message on the queue is one fast write. The wait, when there is one, lives between hops, not inside them. Switch the scenario and watch the bar change.

      Try it: size the service yourself

      Move the sliders and watch the numbers change. This is the napkin math from step 2, live. The readouts update as you drag, and so does the budget bar above.

      messages/day = users × messages per user per day · avg QPS = messages/day ÷ 86,400 · peak QPS = avg QPS × 5

      deliveries/day = messages/day × fan-out · storage/day = deliveries/day × 2 KB

      senders = peak QPS ÷ 500, rounded up · partitions = peak QPS ÷ 1,000, rounded up · shards = (peak QPS × fan-out) ÷ 2,000, rounded up

      fan-out step latency = 120 ms + 10 ms × (fan-out − 10), minimum 120 ms, maximum 600 ms. Every other budget step stays at its normal-scenario value.

      5,000,000messages / day
      ~58average QPS
      ~290peak QPS
      50,000,000deliveries / day
      ~100 GBstorage / day
      1sender instances
      1queue partitions
      2database shards

      Fits the normal design

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic four times. First the two questions, then the HLD that holds, then the LLD that survives, then the gap between them.

      Basic — Two questions, in order

      Definition: Every design splits into two questions. The first is the big picture. What are we building, how big is it, what breaks first. That is the high-level design. The second is the small picture. How does this one piece actually work, line by line, edge case by edge case. That is the low-level design. I have watched strong engineers fail interviews by jumping to the second question before the first one had an answer. HLD first, then LLD, in that order.

      Types:

      • HLD: requirements, napkin math, components, data model, failure list.
      • LLD: one function, its invariants, its edge cases, its tests.

      Uses: I use it for a notification service in this lesson, and for every design interview I have sat in. The HLD says what the system does and how big it is. The LLD says how the sender works, send by send. I use the split in real design reviews too, to stop a two-hour argument about a database index when nobody has agreed what the system does yet.

      Dimensions: A small problem is four boxes: an API, a queue, a sender, a database. The cost is thinking time before coding time, and the trade is speed against rework. An LLD built on a wrong HLD is work you throw away. The HLD is cheap to rewrite. The LLD is not.

      Intermediate — Writing an HLD that holds

      Definition: The rule I use in reviews is simple. No LLD discussion until the HLD has three things written down. The requirements list, the napkin math, and the failure list. If any of the three is missing, the review stops and we go back. That sounds strict. It saves more time than it costs. The napkin math says how big the system is. Two hundred thousand orders a day, three notifications per order, about 7 messages a second on average, about 35 at peak. Tiny. One sender pool, one queue, one database shard, and I say so out loud, because a small answer is a correct answer when the math says small.

      Types:

      • Requirements: what the system must do, and what it must never do. A user must never get the same notification twice.
      • Napkin math: the rough sizing that says whether the answer is one box or fifty.
      • Failure list: what breaks first, and what happens then. Kill the sender mid-send. Duplicate the message in the queue. Expire the lease early.
      • Data model: the tables, the keys, and the invariants they must hold.

      Uses: I use this in every design review I run. I ask for the three things before anyone draws a component. I use the napkin math to stop oversizing, because the engineer who sizes a 7-requests-a-second system for ten thousand a second is building a system nobody asked for.

      Dimensions: At ten times the scale, the HLD changes and the LLD barely changes. More shards, more partitions, a second region. The sender's LLD stays the same: the same function, the same invariants, the same idempotency key. That is the test of a good split. The part that changes with scale and the part that does not should be different parts. The cost is review time and the discipline to stop a review that skipped the HLD. The trade is a slower start against a faster finish.

      Advanced — Writing an LLD that survives

      Definition: The LLD is where I ask for the test list. Not unit tests. The failure tests. The sender in this lesson checks the idempotency key before it sends, not after. That one ordering decision is the difference between a retry and a duplicate. A retry that finds the key returns ALREADY_SENT. A lease claim that fails returns RETRY_LATER, because another worker owns the message. An UNKNOWN result from the provider returns CHECK_FIRST. It never resends blind.

      Types:

      • Idempotency check before send: the key is message id plus recipient id, checked before any send.
      • Lease claims: a worker claims a message for 30 seconds. Another worker cannot take it while the lease holds.
      • UNKNOWN-state handling: the provider may have sent it or not. The sender checks first, it does not guess.

      Uses: I use this when a deploy doubles the sender replica count and the queue redelivers messages whose lease expired during a slow provider call. Without an idempotency key, a retry is a fresh send, and one in twenty users gets the same payment reminder three times. I use it in reviews when I ask for the failure tests by name. If the engineer cannot name the test for an edge case, the edge case is not handled, it is just written down.

      Dimensions: A duplicate rate of one in twenty sounds small until it is 40,000 duplicate sends over nine hours, which is what the incident in this lesson counted. The failure mode is a lease that expires during a slow call, so two workers send the same message. The cost is the dedupe store and the check on every send. The trade is at-least-once delivery with dedupe against exactly-once, which the network will not give you. I take at-least-once and dedupe, because it is honest.

      Master — Reviews and the gap between HLD and LLD

      Definition: The gap between the HLD and the LLD is where real systems break. The HLD for the notification service in this lesson had a no-duplicates requirement written in step one. The LLD never turned it into a key. That gap is the whole lesson. At this level I am not writing designs. I am running the review that catches the gap. The idempotency key becomes a required field in every sender design review, not an optional one. The deploy checklist gets a line: any replica count change gets a duplicate-rate dashboard check for the next hour.

      Types:

      • Failure-test lists: the LLD names the test for every edge case, and the review checks the list, not just the code.
      • Duplicate-rate dashboards: the number that says whether the dedupe is working, watched for an hour after every replica change.
      • Review checklists: the HLD requirements, each traced to the LLD piece that enforces it, so a requirement with no enforcing piece is visible.

      Uses: I use this when I am the reviewer, not the author, and my job is to find the requirement that has no code behind it. I use the runbook rule from this lesson: for a send-volume spike, check the duplicate rate before checking the provider. I use the checklist when a deploy changes the replica count, because that is the change that turned a working sender into a duplicate sender.

      Dimensions: The cost of catching the gap in review is an hour of review time. The cost of catching it in production is a week of apology replies from support, which is what the duplicate incident cost. The failure mode is a review that checks the code and never checks the trace from requirement to code. The trade is thorough review against shipping speed, and I take the review, because a requirement with no enforcing piece is a promise the system will break.

      Go deeper

      The HLD and LLD split is not just an interview trick. It is how I keep a real design review from turning into a two-hour argument about a database index when nobody has agreed what the system does yet. The rule I use in reviews: no LLD discussion until the HLD has three things written down. The requirements list, the napkin math, and the failure list. If any of the three is missing, the review stops and we go back. That sounds strict. It saves more time than it costs, because an LLD built on a wrong HLD is work you throw away. At ten times the scale, the split matters more, not less. The HLD changes: more shards, more partitions, a second region. The sender's LLD barely changes: the same function, the same invariants, the same idempotency key. That is the test of a good split. The part that changes with scale and the part that does not should be different parts. One more thing from real reviews: the LLD is where I ask for the test list. Not unit tests. The failure tests. Kill the sender mid-send. Duplicate the message in the queue. Expire the lease early. If the engineer cannot name the test for an edge case, the edge case is not handled, it is just written down.

      function send(message):
          key = message.id + ":" + message.recipient_id
          if deliveries.exists(idempotency_key = key):
              return ALREADY_SENT        # retry, not a new send
          claim = queue.claim(message.id, lease_seconds = 30)
          if not claim:
              return RETRY_LATER        # another worker owns it
          result = provider.send(message)
          if result == OK:
              deliveries.insert(key, status = DELIVERED)
              return DELIVERED
          if result == UNKNOWN:
              return CHECK_FIRST        # do not resend blind
          return RETRYABLE              # backoff, then try again

      The check comes before the send, not after. That one ordering decision is the difference between a retry and a duplicate.

      One time it broke

      A notification service I worked on started sending the same alert three times to a slice of users. Not all users. About one in twenty. The alerts were payment reminders, so people were getting told three times that their payment had failed. Support tickets piled up, and the users who got three copies assumed the worst: that they had been charged three times. They had not been. The money was fine. The trust was not.

      Timeline

      • 02:14An on-call alert fires for a spike in provider send volume. Nothing is down, so it sits for twenty minutes.
      • 02:36Support reports users getting duplicate payment reminders. The sender logs show each message sent once per worker. Three workers each logged it once.
      • 03:05I find it. A deploy earlier that day doubled the sender replica count, and the queue redelivered messages whose lease expired during a slow provider call. No idempotency key existed. No dedupe check existed. A retry was a fresh send.
      • 03:40A hotfix adds a unique key on message id plus recipient id and a check before every send. Duplicates stop within minutes.
      • Next morningA backfill script counts the damage. Roughly 40,000 duplicate sends over nine hours. No double charges, but a week of apology replies from support.

      What changed after

      Three things. First, the idempotency key became a required field in every sender design review, not an optional one. Second, the deploy checklist got a line: any replica count change gets a duplicate-rate dashboard check for the next hour. Third, the on-call runbook got a new first step for a send-volume spike: check the duplicate rate before checking the provider. The HLD for that service had a no-duplicates requirement written in step 1. The LLD never turned it into a key. That gap is the whole lesson.

      If you remember nothing else

      Design exercise

      Design a notification service for a food delivery app. One million users. An order confirmation, a rider assignment, and a delivery update. Notifications go by push and SMS. A user must never get the same notification twice. You have forty-five minutes.

      Clarifying questions to ask

      How many orders a day, and how many notifications does one order produce? What is the latency target — seconds, or a minute? Does SMS fall back to push, or the other way around? What happens when a user has no push token? How long do we keep delivery records?

      Rough sizing

      Say 200,000 orders a day, three notifications per order, fan-out of one per notification. That is 600,000 messages a day, about 7 a second on average, about 35 at peak. Tiny. One sender pool, one queue, one database shard — and say so out loud, because a small answer is a correct answer when the math says small.

      Shape of the answer

      HLD first: requirements, napkin math, four boxes (API, queue, sender, database), data model with a deliveries table, failure list. Then LLD of the sender only: the one function, the idempotency key, the lease, the edge cases. Do not LLD the API or the database. One component, done properly, beats four components done thinly.

      Trade-offs to name

      At-least-once delivery with dedupe versus exactly-once, which the network will not give you. Push first with SMS fallback costs more per message but reaches users with no app open. A ninety-day retention rule keeps storage honest but limits how far back support can look. Name all three out loud.

      Saved on this device, in this browser.
      Next in this series
      Lesson 28 — Event-Driven Architecture