Platform Engineering · Lesson 164

High-Level Design in Depth: Components, Boundaries, and Data Flow

The brief from the last lesson says what the parcel-tracking service must do, how fast, and how big. The high-level design says what it is made of: which components exist, what each one owns, which arrows are synchronous and which go through a queue, and where one failure stops spreading. This lesson follows one courier scan from the van to the customer's screen, hop by hop, and at every hop names the HLD decision that put it there. By the end you can draw an HLD that answers the three questions every reviewer asks first: what owns this data, what happens when this box is slow, and what happens when it is gone.

Difficulty: Intermediate · Know this first: Lesson 163, Lesson 48, Lesson 32

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

This bar is the freshness budget, not a request latency: the time from a scan in the van to the status on the customer's screen, against the 60-second requirement from the brief. It is a worked example, not a measurement, and the shape is the point: on a normal day the queue wait and the replica lag dominate, and both are deliberate.

Try it: size the write path

Move the sliders. The processor pool is the box that sets the lag during a surge. Watch the queue hop in the freshness bar above follow your numbers.

surge rate = normal peak × multiplier · drain rate = workers × 250 scans/s · backlog growth/s = surge rate − drain rate (never below zero)

queue lag after a 2-minute surge = backlog ÷ drain rate · workers needed = surge rate ÷ 250, rounded up

45,000scans / s in the surge
2,000scans / s drained
5,160,000backlog after 2 minutes
~2,580 squeue lag
180workers needed

The surge breaks the 60-second freshness budget

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

High-level design looks like drawing boxes. It is four levels of the same job: first boxes and arrows that match the requirements, then paths and boundaries that survive load, then failure domains and consistency choices that survive outages, then an HLD that keeps working as the company and the traffic grow around it.

Basic — Boxes, arrows, and one owner per box

At the basic level an HLD is a diagram where every box is a component you will deploy or depend on, and every arrow is a call or a message. The test is simple: each functional requirement from the brief can be traced as a path through the boxes, and each box has one sentence saying what it owns.

Types:

  • Clients: browsers, mobile apps, the courier app, other services that call you.
  • Entry points: gateway, load balancer, API services.
  • Stateful components: databases, caches, queues, object stores.
  • Workers: processors and jobs that do work off the request path.

Uses: the first design review of any service. The win is a shared picture: everyone argues about the same boxes.

Where it breaks: boxes named after technologies, not responsibilities. "Postgres" says nothing about what the box owns; "parcel status, system of record" does.

Try it: Basic

Which label makes a better HLD box?

Now do it: Redraw one diagram you own with every box labelled by what it owns, not what it runs on.

Intermediate — Read path, write path, sync and async

At the intermediate level you separate the paths and mark every arrow. Reads and writes usually have different volumes, latency targets, and failure costs, so they get different paths. Every arrow is either synchronous (the caller waits) or asynchronous (the caller hands off to a queue or event), and the choice is written on the arrow.

Types:

  • Synchronous calls: simple and immediate, and they chain failures and latency.
  • Asynchronous messages: decouple timing and absorb bursts, and add lag and duplicates.
  • Caches and replicas: copies that buy read capacity with staleness.
  • Stateless tiers that scale by adding instances, and stateful tiers that scale by sharding.

Uses: any service where reads and writes differ by 10x or more, or where a client cannot wait. The courier app cannot wait for a database; the customer can wait 20 ms for a cache.

Scale and failure: a synchronous chain of five services at 99.9 percent each is about 99.5 percent available, and its latency is the sum. An async hop breaks both chains and costs you lag and at-least-once delivery, which means idempotency becomes mandatory.

Try it: Intermediate

Why does the ingest API answer 202 Accepted instead of 200 OK after processing?

Now do it: Mark every arrow on one of your diagrams sync or async. Any sync arrow to a slow or bursty component is a candidate for a queue.

Advanced — Failure domains, consistency, and backpressure

At the advanced level the HLD answers what happens when each box is slow or gone. Every component gets a failure row: what breaks, what the user sees, and what takes over. Every copy gets a staleness number. Every queue gets a backpressure plan for when consumers fall behind.

Types:

  • Failure domains: a zone, a region, a dependency, a deploy. The HLD shows which boxes share one.
  • Consistency choices per data item: strong for parcel status in the system of record, eventual for the customer view.
  • Backpressure: queue depth and age alarms, consumer autoscaling, shedding non-critical work first.
  • Degraded modes: serve cached status with a "last updated" time when the replica is down.

Uses: services with an SLO, and anything other teams depend on. Reviewers at this level skip the happy path and read the failure table.

Scale and failure: at 10x the first thing to break is usually the shared dependency nobody drew, such as an auth service or a DNS resolver that every box calls. The cost is redundancy and complexity; the trade is between availability and the number of moving parts a person must understand at 2 AM.

Try it: Advanced

During a surge the scan queue grows for an hour. Nothing is lost. What should the HLD already contain for this case?

Now do it: Add a failure row for every box in one of your HLDs: what breaks, what the user sees, what takes over.

Master — An HLD that keeps working as everything grows

At the master level the HLD is designed for change: more traffic, more teams, more regions, and the migration from today's shape to next year's. You design cells and regions, you draw the cost model next to the component model, and you plan the path between versions so the system never needs a big-bang rewrite.

Types:

  • Cell-based architecture: many identical copies of the stack, each serving a slice of customers, so one bad cell hurts a few percent.
  • Multi-region: active-passive for simplicity, active-active for availability, and the data conflicts active-active brings.
  • Cost model: cost per thousand scans and per thousand reads, per component, so growth has a price tag.
  • Evolution plan: strangler migrations, dual writes with reconciliation, and a version number on the HLD itself.

Uses: platform services, high-growth products, and any system whose outage makes the news. The habit at this level is to ask "what does this design look like at 10x, and how do we get there without stopping?"

Trade-off: cells and regions multiply operational load and make cross-cell queries hard. They are justified by the blast radius they remove, which the master-level HLD states as a number.

Try it: Master

Leadership wants the tracking service to survive a full region outage. What does a master-level HLD add first?

Now do it: For a system you know, write the blast radius of its biggest failure domain as a percentage of customers. Then sketch the cell or region split that would halve it.

Go deeper

The diagram is the start of an HLD, not the whole of it. I pair it with a component table, because a table can be checked and a diagram can only be admired. The table has one row per box and five columns: owns, called by, sync or async, staleness, and failure behaviour.

component       owns                       in from        kind   stale  if down
courier app     local scans, retries       courier        -      -      stores offline, retries
ingest API      validation, auth, accept   courier app    sync   -      app queues scans on phone
scan queue      ordered scans per parcel   ingest API     async  ~2 s   API rejects, phone retries
processor       status rule, dedupe        scan queue     async  -      queue grows, alarm on age
primary DB      scan_history, status (SoR) processor      sync   0      processing stops, no loss
read replica    copy of status             primary        async  ~2 s   reads fail over to cache
status cache    copy, 30 s TTL             read API       sync   30 s   reads go to replica
read API        customer status view       customer       sync   -      page shows last known

Every row has an owner and a failure behaviour. An empty cell in the last column is the design review's first question.

Compute the availability of a path

A synchronous path is only as available as the product of its parts. This short script makes that visible, and shows why the async boundary matters: the courier's write path only depends on the boxes before the queue.

# python3 path_availability.py
from math import prod
avail = {"gateway": 0.9995, "ingest_api": 0.999, "queue": 0.9999,
         "processor": 0.999, "primary_db": 0.9995}
sync_write = ["gateway", "ingest_api", "primary_db"]        # no queue
async_write = ["gateway", "ingest_api", "queue"]            # courier waits for these only
for name, path in [("sync", sync_write), ("async", async_write)]:
    a = prod(avail[c] for c in path)
    print(f"{name:5} {a:.5f}  ~{(1 - a) * 43200:.0f} min/month failing")

Expected: the sync path is about 99.80 percent, roughly 86 failing minutes a month from the courier's point of view; the async path is about 99.84 percent and, more importantly, no longer depends on the database at all. Add "processor" to the sync list and run it again to see the cost of one more synchronous hop.

One time it broke

A tracking service I reviewed wrote courier scans synchronously to the database. It had been fine for a year. Then a routine index rebuild slowed writes from 20 ms to four seconds for forty minutes on a Monday morning, the busiest scanning hour of the week.

Timeline

  • 08:05Index rebuild starts, scheduled for "low traffic". Nobody had checked when couriers scan.
  • 08:09Courier apps begin timing out at three seconds and retrying. Each retry is a new write against the same slow table.
  • 08:20The connection pool saturates. Read requests now queue behind writes, and the customer page slows too.
  • 08:45The rebuild finishes. Writes recover. About 30,000 scans were stored twice, and history pages show duplicates for a week.

What changed after

The redesign put a queue behind the ingest API, moved the status rule into one processor, made scan id a unique key, and split the read path into its own deployment with its own connection pool. The next index rebuild slowed processing for an hour and nobody outside the team noticed. The HLD review checklist gained one line: every write path that a client retries must end at a durable buffer and an idempotency key.

If you remember nothing else

Design exercise

Draw the HLD for a ride receipt service: when a ride ends, the rider gets a receipt in the app and by email within two minutes, and finance can query every receipt for seven years.

Clarifying questions to ask

How many rides end per second at peak? Does the receipt need the final fare from the payment system, and how long can that take? Are receipts ever corrected after they are sent? What does finance query by: rider, date, city? Is email delivery part of the requirement or best effort?

Rough sizing

Say 50 million rides a month, peaking at about 60 a second on a Friday night with a 3x surge: 180 receipts a second. At 3 KB each that is about 150 GB a month and 13 TB over seven years before compression, so the hot store and the archive are different boxes.

Shape of the answer

Ride-ended event onto a queue; a receipt builder that waits for the fare-finalised event and owns the receipt format; a receipts table as system of record; an email sender behind its own queue with retries; a read API for the app; nightly export to a columnar archive for finance. Two async boundaries: fare finalisation and email.

Trade-offs to name

Waiting for the final fare delays the receipt but avoids sending corrections; sending immediately with "pending" is faster and noisier. One store for app and finance is simpler; a separate archive keeps the app fast and makes seven-year queries cheap. Email inside the builder is fewer boxes; email behind its own queue keeps a slow email provider from delaying in-app receipts.

Saved on this device, in this browser.
Next in this series · Lesson 165
Data Modeling and Schema Design: Entities, Access Patterns, Keys, and Migrations

The HLD says the primary database is the system of record. It does not say what is in it. The data model is the part of a design that outlives every other part: services get rewritten, queues get swapped, but the tables and their…

Go to Lesson 165 →

Or browse the full course from the Lessons index.