Every design interview I have sat in, and every real system I have built, splits into two questions. The first is the big picture: what are we building, how big is it, what breaks first. The second is the small picture: how does this one piece actually work, line by line, edge case by edge case. The first is the high-level design. The second is the low-level design. I have watched strong engineers fail interviews by jumping to the second question before the first one had an answer. This lesson walks one small problem, a notification service, through both. HLD first, then LLD, in that order, and I will show you what each one has to prove before you move on.
Difficulty: Advanced · Know this first: Lesson 02, Lesson 03
Extra diagram · Architecture map
One trigger, one sender, one delivery. I add the queue and the database, then the four workers and the fallback.
Extra diagram · Second trip
The design itself is the trip here. I follow the requirements as they become boxes, then a contract, then checked edge cases.
Each hop owns a slice of the end-to-end latency budget for the notification service. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: fan-out eats most of the budget, because one message becomes many deliveries. The trigger and the queue take no slice of the bar. The trigger is an event, not work, and putting a message on the queue is one fast write. The wait, when there is one, lives between hops, not inside them. Switch the scenario and watch the bar change.
Move the sliders and watch the numbers change. This is the napkin math from step 2, live. The readouts update as you drag, and so does the budget bar above.
messages/day = users × messages per user per day · avg QPS = messages/day ÷ 86,400 · peak QPS = avg QPS × 5
deliveries/day = messages/day × fan-out · storage/day = deliveries/day × 2 KB
senders = peak QPS ÷ 500, rounded up · partitions = peak QPS ÷ 1,000, rounded up · shards = (peak QPS × fan-out) ÷ 2,000, rounded up
fan-out step latency = 120 ms + 10 ms × (fan-out − 10), minimum 120 ms, maximum 600 ms. Every other budget step stays at its normal-scenario value.
Fits the normal design
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times. First the two questions, then the HLD that holds, then the LLD that survives, then the gap between them.
Definition: Every design splits into two questions. The first is the big picture. What are we building, how big is it, what breaks first. That is the high-level design. The second is the small picture. How does this one piece actually work, line by line, edge case by edge case. That is the low-level design. I have watched strong engineers fail interviews by jumping to the second question before the first one had an answer. HLD first, then LLD, in that order.
Types:
Uses: I use it for a notification service in this lesson, and for every design interview I have sat in. The HLD says what the system does and how big it is. The LLD says how the sender works, send by send. I use the split in real design reviews too, to stop a two-hour argument about a database index when nobody has agreed what the system does yet.
Dimensions: A small problem is four boxes: an API, a queue, a sender, a database. The cost is thinking time before coding time, and the trade is speed against rework. An LLD built on a wrong HLD is work you throw away. The HLD is cheap to rewrite. The LLD is not.
Definition: The rule I use in reviews is simple. No LLD discussion until the HLD has three things written down. The requirements list, the napkin math, and the failure list. If any of the three is missing, the review stops and we go back. That sounds strict. It saves more time than it costs. The napkin math says how big the system is. Two hundred thousand orders a day, three notifications per order, about 7 messages a second on average, about 35 at peak. Tiny. One sender pool, one queue, one database shard, and I say so out loud, because a small answer is a correct answer when the math says small.
Types:
Uses: I use this in every design review I run. I ask for the three things before anyone draws a component. I use the napkin math to stop oversizing, because the engineer who sizes a 7-requests-a-second system for ten thousand a second is building a system nobody asked for.
Dimensions: At ten times the scale, the HLD changes and the LLD barely changes. More shards, more partitions, a second region. The sender's LLD stays the same: the same function, the same invariants, the same idempotency key. That is the test of a good split. The part that changes with scale and the part that does not should be different parts. The cost is review time and the discipline to stop a review that skipped the HLD. The trade is a slower start against a faster finish.
Definition: The LLD is where I ask for the test list. Not unit tests. The failure tests. The sender in this lesson checks the idempotency key before it sends, not after. That one ordering decision is the difference between a retry and a duplicate. A retry that finds the key returns ALREADY_SENT. A lease claim that fails returns RETRY_LATER, because another worker owns the message. An UNKNOWN result from the provider returns CHECK_FIRST. It never resends blind.
Types:
Uses: I use this when a deploy doubles the sender replica count and the queue redelivers messages whose lease expired during a slow provider call. Without an idempotency key, a retry is a fresh send, and one in twenty users gets the same payment reminder three times. I use it in reviews when I ask for the failure tests by name. If the engineer cannot name the test for an edge case, the edge case is not handled, it is just written down.
Dimensions: A duplicate rate of one in twenty sounds small until it is 40,000 duplicate sends over nine hours, which is what the incident in this lesson counted. The failure mode is a lease that expires during a slow call, so two workers send the same message. The cost is the dedupe store and the check on every send. The trade is at-least-once delivery with dedupe against exactly-once, which the network will not give you. I take at-least-once and dedupe, because it is honest.
Definition: The gap between the HLD and the LLD is where real systems break. The HLD for the notification service in this lesson had a no-duplicates requirement written in step one. The LLD never turned it into a key. That gap is the whole lesson. At this level I am not writing designs. I am running the review that catches the gap. The idempotency key becomes a required field in every sender design review, not an optional one. The deploy checklist gets a line: any replica count change gets a duplicate-rate dashboard check for the next hour.
Types:
Uses: I use this when I am the reviewer, not the author, and my job is to find the requirement that has no code behind it. I use the runbook rule from this lesson: for a send-volume spike, check the duplicate rate before checking the provider. I use the checklist when a deploy changes the replica count, because that is the change that turned a working sender into a duplicate sender.
Dimensions: The cost of catching the gap in review is an hour of review time. The cost of catching it in production is a week of apology replies from support, which is what the duplicate incident cost. The failure mode is a review that checks the code and never checks the trace from requirement to code. The trade is thorough review against shipping speed, and I take the review, because a requirement with no enforcing piece is a promise the system will break.
The HLD and LLD split is not just an interview trick. It is how I keep a real design review from turning into a two-hour argument about a database index when nobody has agreed what the system does yet. The rule I use in reviews: no LLD discussion until the HLD has three things written down. The requirements list, the napkin math, and the failure list. If any of the three is missing, the review stops and we go back. That sounds strict. It saves more time than it costs, because an LLD built on a wrong HLD is work you throw away. At ten times the scale, the split matters more, not less. The HLD changes: more shards, more partitions, a second region. The sender's LLD barely changes: the same function, the same invariants, the same idempotency key. That is the test of a good split. The part that changes with scale and the part that does not should be different parts. One more thing from real reviews: the LLD is where I ask for the test list. Not unit tests. The failure tests. Kill the sender mid-send. Duplicate the message in the queue. Expire the lease early. If the engineer cannot name the test for an edge case, the edge case is not handled, it is just written down.
function send(message):
key = message.id + ":" + message.recipient_id
if deliveries.exists(idempotency_key = key):
return ALREADY_SENT # retry, not a new send
claim = queue.claim(message.id, lease_seconds = 30)
if not claim:
return RETRY_LATER # another worker owns it
result = provider.send(message)
if result == OK:
deliveries.insert(key, status = DELIVERED)
return DELIVERED
if result == UNKNOWN:
return CHECK_FIRST # do not resend blind
return RETRYABLE # backoff, then try againThe check comes before the send, not after. That one ordering decision is the difference between a retry and a duplicate.
A notification service I worked on started sending the same alert three times to a slice of users. Not all users. About one in twenty. The alerts were payment reminders, so people were getting told three times that their payment had failed. Support tickets piled up, and the users who got three copies assumed the worst: that they had been charged three times. They had not been. The money was fine. The trust was not.
Three things. First, the idempotency key became a required field in every sender design review, not an optional one. Second, the deploy checklist got a line: any replica count change gets a duplicate-rate dashboard check for the next hour. Third, the on-call runbook got a new first step for a send-volume spike: check the duplicate rate before checking the provider. The HLD for that service had a no-duplicates requirement written in step 1. The LLD never turned it into a key. That gap is the whole lesson.
Design a notification service for a food delivery app. One million users. An order confirmation, a rider assignment, and a delivery update. Notifications go by push and SMS. A user must never get the same notification twice. You have forty-five minutes.
How many orders a day, and how many notifications does one order produce? What is the latency target — seconds, or a minute? Does SMS fall back to push, or the other way around? What happens when a user has no push token? How long do we keep delivery records?
Say 200,000 orders a day, three notifications per order, fan-out of one per notification. That is 600,000 messages a day, about 7 a second on average, about 35 at peak. Tiny. One sender pool, one queue, one database shard — and say so out loud, because a small answer is a correct answer when the math says small.
HLD first: requirements, napkin math, four boxes (API, queue, sender, database), data model with a deliveries table, failure list. Then LLD of the sender only: the one function, the idempotency key, the lease, the edge cases. Do not LLD the API or the database. One component, done properly, beats four components done thinly.
At-least-once delivery with dedupe versus exactly-once, which the network will not give you. Push first with SMS fallback costs more per message but reaches users with no app open. A ninety-day retention rule keeps storage honest but limits how far back support can look. Name all three out loud.