Platform Engineering · Lesson F19

TCP and UDP: How Messages Actually Travel

An IP address gets your message to the right machine. It does not get it to the right program on that machine, and it does not promise the message arrives at all. That is the job of the transport layer. This lesson walks a message from your laptop to a server, packet by packet, and shows why the web uses TCP and a video call uses UDP.

Difficulty: Beginner · Know this first: F18 — The OSI Model: Seven Layers

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenarios, not a measurement from a live system. The shape is honest: on TCP the handshake and the network carry the cost, and on a blocked port the timeout swallows everything else. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

Ports: the doors on one address

Start with one machine. Add one door at a time until you can see which programs answer, and which door stays shut.

4 well known ports1 closed door1 address, many programs
Your laptop
sends the packet
Router
the default gateway
Firewall
reads the rules
Port 80
plain web, HTTP
Port 443
secure web, HTTPS
Port 22
SSH login
Port 53
DNS questions
Closed door
port 8080, no rule
Request / flowResponse · the way backFailure / dead end

Extra diagram · from scratch to master

TCP shakes hands, UDP just talks

Start with two machines and nothing between them. Add the handshake, the data, and the acknowledgements until you can see the trade.

3 handshake packets1 lost packet2 protocols, one road
Sender
your laptop
SYN
I want to talk
SYN-ACK
I hear you
ACK
let us start
Data packets
numbered 1, 2, 3
ACKs back
got 1, got 2
Receiver
the server
Lost packet
packet 2, gone
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch it bend

Pick a job and a lossy network. The simulator sends the packets, drops some, and shows you what each protocol does about it. The numbers are a teaching example, not live measurements. The shape is the lesson: TCP pays in time, UDP pays in missing pieces.

Web page

The job decides how many packets there are and which protocol fits. A web page must arrive whole. A voice call must arrive now.

10

Share of packets the network drops. 0 percent is a clean network. 20 percent is a bad mobile connection.

Try this: (1) Web page, 10 percent loss. TCP loses 4 packets, resends 4, delivers all 40, and pays 200 ms extra. UDP loses 4 and delivers 36. (2) Voice call, 10 percent loss. UDP loses 5 packets and the call goes on with a glitch. TCP would resend them and the voice would arrive late, which is worse. (3) DNS lookup, any loss. One packet, so 10 percent loss means it usually arrives, and when it does not, your machine just asks again.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same idea four times here. First I learn what it is and why it exists. Then I learn to use it and read what it tells me. Then I design with it so one failure cannot take the system down. Last I run it for the world.

Basic — What a packet and a port are

Definition: A packet is a small envelope carrying a piece of a message plus the addresses it needs. A port is a number that names the program on the machine, like an apartment number on a street address. TCP is the careful protocol that shakes hands and resends what is lost. UDP is the fast protocol that sends and moves on.

Types:

  • TCP for web pages, files, and anything that must arrive whole.
  • UDP for voice, video, games, and DNS, where speed beats completeness.
  • Ports 80, 443, 22, and 53 as the doors you will meet first.

Uses: I use this when a connection fails. I ask which port it aimed at, whether the handshake finished, and whether packets are being lost, in that order.

Dimensions: At low traffic everything works and the difference between TCP and UDP is invisible. What breaks first is a blocked port, and I notice it as a timeout, not an error. It costs nothing to learn the four ports. It costs hours to debug a connection without them. The trade is simple: TCP buys certainty with time, UDP buys speed with gaps.

Intermediate — Handshakes, sequence numbers, retransmission

Definition: TCP numbers every packet and acknowledges what arrives. A gap in the numbers means a lost packet, and TCP resends just that packet. The window size says how much unacknowledged data the sender may have in flight. The firewall sits in the middle and allows or drops packets by port. DHCP hands out the IP address, the default gateway, and the DNS server, and the MAC address names the network card on the local wire.

Types:

  • Three-way handshake: SYN, SYN-ACK, ACK.
  • Retransmission of only the missing packets.
  • Firewalls and security groups as rule lists by port.

Uses: I use packet captures at this level. I read the handshake, count the retransmissions, and tell a network problem apart from an app problem before I touch any code.

Dimensions: As loss rises, TCP slows down on purpose, because loss looks like congestion. What breaks is the assumption that the network is clean, and I notice it in retransmission counts before users notice the slowness. It costs a capture tool and the habit of reading it. The trade is that reliability costs round trips, and on a long or lossy path those round trips add up.

Advanced — Congestion, firewalls, and NAT

Definition: TCP watches for loss and shrinks its window to ease congestion, then grows it again when the road is clear. A firewall rule list is read top to bottom and the first match wins. NAT sits at the edge of a private network and rewrites addresses so many private machines share one public address, using ports to tell the replies apart.

Types:

  • Congestion windows that grow and shrink with the network.
  • Stateful firewalls that remember a connection once it is allowed.
  • NAT, where the port number is the return address.

Uses: I use this when I design a service that must be reachable from the internet and safe at the same time: open one port, close the rest, and let the firewall log what knocked.

Dimensions: At ten times the traffic the firewall rule list and the connection table become the narrow doors. What breaks is the state table filling up, and I notice it as new connections failing while old ones work. It costs careful rule hygiene and monitoring of the firewall itself. The trade is that every open port is a door, and every door is a risk I must be able to explain.

Master — Tuning the transport for the world

Definition: At this level I tune TCP for the path it runs on: window sizes for long fat pipes, keep-alive for connections that sit idle, and timeouts matched to the real round trip. I know when TCP is the wrong tool and UDP with my own reliability on top, or a protocol like QUIC that runs over UDP, fits better. I run services where a lost packet is a metric with an alert, not a mystery.

Types:

  • TCP tuning per path: window scaling, keep-alive, timeouts.
  • QUIC and HTTP/3, which move the handshake into UDP.
  • Custom reliability on UDP for games and live media.

Uses: I use this when a service crosses continents or runs on lossy mobile networks, where the default TCP settings waste time on every round trip.

Dimensions: At a hundred times the traffic the transport is a system of its own, with per-region tuning and real loss budgets. What breaks is a tuning choice made for one path and applied to all of them, and I notice it in p99 latency on one region only. It costs measurement on every path and the discipline to tune per path. The trade is that the defaults are safe everywhere and fast nowhere in particular.

Go deeper

Sequence numbers are the quiet hero of TCP. Every byte gets a number, so the receiver can spot a gap, sort out-of-order packets, and ask for exactly the piece that is missing. Without them, TCP would have to resend whole messages, and the web would be far slower than it is.

The window is the second hero. It says how much data the sender may have in flight before it must wait for an acknowledgement. A big window fills a fast pipe. A small window starves it. When packets go missing, TCP shrinks the window, because loss usually means the network is crowded and sending more would make it worse.

NAT is where ports do double duty. A home router gives every device a private address, then rewrites them to one public address on the way out. It tells the replies apart by port number. That is why a server on your home network is not reachable from the internet unless you forward a port through the router.

QUIC is the new arrival worth knowing by name. It runs over UDP, folds the handshake and the encryption together, and starts sending data sooner than TCP plus TLS can. HTTP/3 is HTTP over QUIC. It does not replace TCP everywhere, but on lossy mobile networks it is the reason some pages feel faster.

At 10x the traffic, the shape holds and the numbers grow. The handshake stays one round trip. The network time grows with distance and load. I add capacity at the firewall and the load balancer first, because those are the doors every packet passes through.

At 100x, the design changes. I put the service close to the user so the round trip shrinks, and I terminate the connection at the edge. The inside traffic stays on private networks, and the firewall rules become a reviewed list with owners, not a file nobody dares to touch.

# show every listening port on this machine
ss -tuln

# example output (teaching example)
# tcp  0.0.0.0:22    listening   (SSH)
# tcp  0.0.0.0:443   listening   (web, encrypted)
# udp  0.0.0.0:53    listening   (DNS)

What the snippet means: ss lists the sockets on a machine. The t flag is TCP, u is UDP, l is listening, n is numeric so it shows port numbers instead of names. Each line is a door that is open on this machine. If the port my service needs is not in this list, the program is not listening, and no firewall rule in the world will fix that.

One time it broke

What broke

A small internal tool stopped loading for everyone in the office. The page just spun and then timed out. The tool had not changed. The server was up. I could log in to the server over SSH and the tool answered fine from inside the machine. From my laptop, nothing.

Timeline

  • 09:12,the first report came in: the tool page times out.
  • 09:20,I opened the page myself and got the same spin, then a timeout. No error page, just waiting.
  • 09:26,I logged in to the server over SSH, port 22. That worked. From inside, the tool answered in milliseconds.
  • 09:35,I checked the listening ports. The tool was listening on port 8080, as always.
  • 09:41,I checked the firewall rules. A cleanup the night before had removed the allow rule for port 8080. Port 22 and 443 were still there.
  • 09:47,I added the rule back. The page loaded immediately.
  • 10:05,I confirmed from three machines on different networks, not just mine.

What changed after

Three things. First, firewall rules got the same review as code: no rule is removed without naming the service that uses the port. Second, a check that connects to each internal tool from outside the server every five minutes, so a missing rule pages us instead of waiting for a user to report it. Third, I wrote down the three questions in order: is the port open, did the handshake finish, are packets being lost. A timeout means I start at the firewall, not at the app.

If you remember nothing else

Design exercise

Design message delivery for a chat app with one million users. Text messages must never be lost. Typing indicators (she is typing) can be lost. Voice notes must arrive whole. Live voice calls must feel instant.

Clarifying questions to ask

  • Which parts go over TCP and which over UDP, and what breaks if you pick wrong for each one?
  • What port does the chat server listen on, and what does the firewall allow through?
  • A user on a bad mobile connection loses 10 percent of packets. What does each part of the app feel like?
  • What happens when the chat server moves to a new port and the firewall rule is not updated?
  • How do you tell a lost packet apart from a dead server, from the user's side?

Rough sizing

Start with the counts. One million users, each sending about 10 messages an hour, is about 2,800 messages a second. Each message is a few packets. Text and voice notes go over TCP, so every lost packet costs a retransmission of about 50 ms on a typical path. Typing indicators and live calls go over UDP, so loss shows up as a missing indicator or a voice glitch, never as delay. I size the firewall connection table for the peak, not the average, because every TCP connection is a row in that table.

Trade-offs to name

TCP for text buys certainty and costs latency when the network is lossy. UDP for calls buys low delay and costs the odd glitch. Opening one port keeps the firewall simple and makes the chat server a single point of failure, so I run more than one server behind a load balancer. And the timeout rule applies here too: a user who sees a spinning send button is usually looking at a dropped packet, not a broken app.

Next in this series
Lesson F20 — DNS: Names to Numbers