System Design · Lesson 46

Networking Deep Dive

Lesson 01 walked the request from the browser to the database. This lesson opens up the first three hops and looks inside them. A name becomes an address. A connection opens. A handshake proves the server is who it says it is. Then the request itself finally moves. I have spent whole on-call nights inside these three hops, because a slow page is often not a slow app. It is a DNS answer that took a walk around the world, or a handshake that ran twice, or a connection that got thrown away after one request. This lesson follows one request at the packet level, counts the round trips, and shows you where the time actually goes.

Difficulty: Advanced · Know this first: Lesson 01, Lesson 02, Lesson 31
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one request, not a measurement from a live system, but the shape is honest: on a fresh connection, the setup hops cost more than the app work. DNS, TCP, and TLS together are 80 ms of the 165 ms normal trip. The app is 35 ms. Step through the lesson or switch the scenario and watch the bar change. The cold start nearly doubles the trip without the app doing anything different. The reused connection cuts it to less than half, because three hops drop to zero.

Change the network and watch the handshake bill

The budget above is one fixed trip. This box is yours. Push the round trip up, switch to TLS 1.2, turn keep-alive off, and watch the handshake bill grow. Then ask the question I ask before every deploy: how much of this page load is setup, and how much is the app?

How long one round trip takes on this network. A local user is near 10 ms. A user on the other side of the world is near 200 ms. TCP costs one round trip. TLS 1.3 costs one round trip. TLS 1.2 costs two. The request and the response each cost half a round trip.

TLS 1.3 finishes its handshake in one round trip. TLS 1.2 needs two. The crypto is not the cost. The extra round trip is.

On means one connection serves all ten requests, so the handshake is paid once. Off means every request opens a fresh connection and pays the handshake again.

Hit means the resolver already knows the address, 20 ms. Miss means the resolver walks the tree from the root servers down, 120 ms. Paid once per page load, not once per request.

Handshake cost60 msTCP + TLS, paid per fresh connection
Per-request cost75 msround trip + 45 ms of fixed hops
Total for 10 requests840 mssetup paid once, then 10 requests
Setup share10%share of the 10-request total spent on setup instead of answers

Try this: (1) Turn keep-alive off at the defaults. The total jumps from 840 ms to 1,470 ms and nothing else changed. (2) Now set TLS to 1.2 and RTT to 200 with keep-alive still off. The handshake alone is 600 ms per request and the total is 8,670 ms. Same app, same code. Only the network changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is one request through nine hops. Production is the rest of it. Here is what I actually spend my time on once the handshakes are counted and the budget is written down.

QUIC and HTTP/3

QUIC moves the transport from TCP to UDP, and HTTP/3 runs on top of it. The reason is the handshake. QUIC folds the transport handshake and the TLS handshake into one exchange, so a fresh connection can send data after a single round trip instead of two or three. It also supports 0-RTT, where a returning client sends data in the very first packet using keys from the last visit. That is fast, and it has a price. A 0-RTT request can be replayed by anyone who captures it, so I never allow 0-RTT on anything that changes state: no logins, no payments, no writes. Reads only. And QUIC breaks in places that assume TCP. Some corporate firewalls block UDP outright, some middleboxes mangle it, and some load balancers still do not speak it. I roll QUIC out behind a flag and watch the failure rate by network before I trust it everywhere.

Head-of-line blocking

Head-of-line blocking is one slow thing holding up everything behind it. TCP has it at the connection level: one lost packet stalls every stream on that connection until the packet is retransmitted. HTTP/2 fixed it at the HTTP level with streams, but the streams still ride one TCP connection, so a lost packet still stalls them all. QUIC fixes it properly, because each stream is independent: a lost packet stalls one stream, not the rest. I think about this when I choose how many connections to open. One connection is efficient until one loss ruins it. Several connections cost more handshakes and buy isolation. There is no free answer, only a trade.

eBPF: watching the network from the kernel

eBPF lets me run small programs inside the kernel and watch the network without changing the app. I can see every TCP retransmit, every DNS lookup time, every connection that took too long to open, per pod, in real time. Before eBPF I guessed at this from logs. Now I measure it at the packet level. The rule I follow is simple: observe first, change second. A retransmit counter that spikes on one node tells me more in a minute than an hour of reading app logs.

MTU and packet sizing

MTU is the biggest packet a link will carry, usually 1500 bytes on the public internet. A packet bigger than that has to be fragmented or dropped. Path MTU discovery is how the two ends agree on the safe size, and it depends on ICMP messages that say a packet was too big. If a firewall blocks those messages, the sender never learns, keeps sending big packets, and they keep dying. That is the black hole: small requests work, big responses stall, and nothing logs an error. Jumbo frames raise the MTU to 9000 inside a private network, which helps east-west traffic between services, but the moment a packet leaves for the internet it hits 1500 again. I keep the public path at 1500, allow the ICMP messages that make discovery work, and set jumbo frames only where I control both ends.

What changes at 10x and 100x

At 10x the traffic, the handshakes are the first thing that breaks. I terminate TLS at the edge, keep connections alive between the edge and the pods, and watch the connection count per pod. At 100x, DNS becomes a design decision, not a config line: anycast DNS, short TTLs on records I might move, and an edge that answers close to the user. The budget stops being a target and becomes a contract, and every hop owns its share of it.

Measure the hops with curl

This one command splits a request into its hops, so you can measure them from your own laptop:

curl -o /dev/null -s -w \
  "dns: %{time_namelookup}s\nconnect: %{time_connect}s\ntls: %{time_appconnect}s\nfirst byte: %{time_starttransfer}s\ntotal: %{time_total}s\n" \
  https://app.example.com/health

Five numbers from one command. DNS time is the name lookup. Connect minus DNS is the TCP handshake. TLS minus connect is the TLS handshake. First byte minus TLS is the server work plus the request travel. Total is the whole trip. I run this before I open a dashboard, because it tells me which hop to blame in ten seconds. If DNS is the big number, I look at the resolver. If TLS is the big number, I look at the version and the round trip. If first byte is the big number, I look at the app. The numbers point. I follow them.

One time it broke

This one stayed with me because nothing failed. Every dashboard was green, every pod was healthy, and the users were staring at a spinner that never stopped.

What broke

After a network change, small pages loaded fine and every large download stalled. API calls under a kilobyte answered in milliseconds. Anything over about 1400 bytes started and never finished. The app was healthy. The pods were healthy. The load balancer was healthy. Users saw a spinner that never stopped, and the app logs showed nothing at all, because the requests that stalled never completed, so there was nothing to log.

Timeline

  • 09:10First ticket: the report download hangs. I try it myself. Small pages fine, the download stuck.
  • 09:25I check the app and the database. Both green. CPU low, no slow queries. The app is not the problem.
  • 09:40I try the same download from inside the cluster. It works. I try it from outside. It stalls. The problem is on the path between the outside and the inside, not in the app.
  • 10:05I run curl with timing output. DNS fast, connect fast, TLS fast, first byte never arrives for large responses. Small responses arrive fine. That split is the clue.
  • 10:30I check the MTU. A new link in the path uses jumbo frames, 9000 bytes. The firewall on that link blocks the ICMP messages that tell the sender a packet was too big. Big packets go out, get dropped, and nobody tells the sender.
  • 11:00I clamp the TCP maximum segment size to fit a 1500 byte MTU on the edge, and the downloads start finishing.
  • 11:20I confirm from three networks, including a mobile connection, that large downloads complete.

What changed after

Three things. First, the firewall now allows the ICMP messages that make path MTU discovery work. Blocking them saved nothing and broke large responses. Second, the edge clamps the segment size to 1500 for anything leaving toward the internet, so a jumbo frame inside the network can never become a black hole outside it. Third, I added a synthetic check that downloads a 2 MB file from outside every five minutes. The old health check only fetched a tiny page, so it never saw this failure. A health check has to look like a real user, and real users download large things.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real network path. Take a minute with it before you read my approach.

The prompt

Design the network path for a login page that must answer in under 200 ms at p99 for users on three continents. The page needs DNS, TLS, a load balancer, and an app that checks a password. You have an edge network, a gateway, app pods, and a session store. Sketch the path and the budget.

Clarifying questions to ask first

I would ask five things before I sized anything. Where are the users, and what is the round trip from each continent? How fresh must the session check be, or can a valid session be answered from the edge? What happens when the password check is slow: do we fail, retry, or queue? Is 200 ms the target for the first visit or for repeat visits, because the handshake budget is very different? And what is the failure we design for: one region down, one edge node down, or the session store slow?

Rough sizing, worked

Start with the budget from this lesson. A user 30 ms away pays DNS 20, TCP 30, TLS 1.3 30, request 15, load balancer 5, app 35, trip back 15. That is 150 ms on a first visit, inside the 200 ms target, with 50 ms of headroom. A user 150 ms away pays DNS 20, TCP 150, TLS 150, request 75, load balancer 5, app 35, trip back 75. That is 510 ms, and it fails before the app does anything. So the far user cannot be served from one region. I put an edge in each continent, terminate TLS at the edge, and answer repeat visits from a kept-alive connection. The far user on a reused connection pays request 75, load balancer 5, app 35, trip back 75: 190 ms, inside the target. The first visit is over budget and I say so honestly, then I fix it with anycast DNS and a 0-RTT resume where reads allow it. Little's Law for the sizing: 5,000 logins a second at 0.15 seconds in the system is 750 requests inside at once, so I plan the pods and the session store connections for 750 plus headroom, and I alert at 80 percent.

Trade-offs to name

Terminating TLS at the edge buys speed and costs end-to-end encryption inside the cluster, so I keep the inside traffic on a private network. Short DNS TTLs buy fast failover and cost extra lookups. Keep-alive buys speed and costs connection slots on the edge, so I set an idle timeout and watch the count. 0-RTT buys a faster first packet and costs replay safety, so I allow it for reads only, never for the login itself. And the target is p99, not the average. An average of 120 ms with a p99 of 600 ms fails this design.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 47 — Containers & Docker
Coming next