Platform Engineering · Lesson F20

DNS: Names to Numbers

You type a name. The network only understands numbers. Something has to translate, every single time, before anything else can happen. That something is DNS, the Domain Name System. This lesson walks the lookup from your laptop to the root of the internet and back, and shows why a name change can take hours to reach everyone.

Difficulty: Beginner · Know this first: F19 — TCP and UDP: How Messages Actually Travel

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenarios, not a measurement from a live system. The shape is honest: a cached answer costs almost nothing, a full walk costs the sum of three servers, and a name that does not exist costs the same walk for no reward. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The walk, step by step

Start with one question. Add one server at a time until the whole walk is on the map, dead end included.

3 servers in the walk1 cache1 dead end
Browser
asks the name
Resolver
takes the question
Root
points to .com
TLD .com
points onward
Authoritative
holds the answer
A record
name to address
Cache
kept for the TTL
NXDOMAIN
no such name
Request / flowResponse · the way backFailure / dead end

Extra diagram · from scratch to master

Record types: four answers DNS can give

Start with one question. Add one record type at a time until you can read what DNS actually returns.

4 record types1 TTL on each1 wrong record
Your question
example.com?
A record
IPv4 address
AAAA record
IPv6 address
CNAME
one name to another
MX record
mail goes here
TTL clock
how long to keep it
Cache
answers from memory
Wrong record
old address, live
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch it bend

Pick a name and run the walk. The simulator asks the root, the TLD, and the authoritative server in order, and shows the record that comes back. The addresses and TTLs are teaching examples, labeled as such. The walk is the lesson.

example.com

Each name walks a different path. The .in name ends at a different TLD, and the last name does not exist at all.

No, cold cache

A warm cache skips the whole walk. That is the difference between 4 ms and 72 ms in this lesson.

Try this: (1) example.com, cold cache. Root points to .com, .com points to the authoritative server, and the A record comes back with a 300 second TTL. (2) Same name, warm cache. Answered from memory in about 4 ms, no walk at all. (3) no-such-name.example.com, cold cache. The walk runs to the authoritative server and the answer is NXDOMAIN: the name does not exist.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same idea four times here. First I learn what it is and why it exists. Then I learn to use it and read what it tells me. Then I design with it so one failure cannot take the system down. Last I run it for the world.

Basic — Names to numbers

Definition: DNS translates a name like example.com into an IP address like 93.184.216.34. A resolver takes your question, checks its cache, and if it has no answer, walks the tree: root, then TLD, then authoritative. The record it brings back says what kind of answer it is, and the TTL says how long it may be kept.

Types:

  • A records for IPv4 addresses.
  • AAAA records for IPv6 addresses.
  • CNAME records for aliases, MX records for mail.

Uses: I use this when a site will not load. I check whether the name resolves at all before I check anything about the server.

Dimensions: At low traffic a cached answer costs a few milliseconds. What breaks first is a stale record, and I notice it when some users reach the new server and some still reach the old one. It costs nothing to learn the walk. It costs hours to debug a site without it. The trade is the TTL: long means fast and slow to fix, short means slower and quick to fix.

Intermediate — Caching, TTLs, and the walk

Definition: DNS is a cache of a cache of a cache: the browser, the operating system, and the resolver each keep answers. The TTL on each record is its own expiry date. A resolver that serves a stale answer is not broken. It is doing exactly what the TTL told it to do.

Types:

  • Browser cache, operating system cache, resolver cache.
  • Positive answers cached by their TTL.
  • Negative answers, NXDOMAIN, cached briefly too.

Uses: I use dig at this level. I ask the authoritative server directly, read the TTL on the answer, and tell a DNS problem apart from a server problem in one command.

Dimensions: As a service grows, the number of names grows with it, and each one needs its TTL chosen on purpose. What breaks is a TTL set once and forgotten, and I notice it during a migration when the old address refuses to die. It costs a naming habit: short TTLs on records I may need to move, long TTLs on records that never move. The trade is that every cache hit saves a walk, and every saved walk is a fix that arrives late.

Advanced — Running DNS for a service

Definition: At this level I run or configure the authoritative servers for a domain: the records, the TTLs, the second nameserver for redundancy. I use short TTLs before a migration and raise them after. I use health checks so a dead server's address stops being served, and I know DNSSEC exists to prove an answer was not tampered with on the way.

Types:

  • Authoritative zones with more than one nameserver.
  • Short TTLs around migrations, long TTLs in steady state.
  • DNS-based failover and load balancing across addresses.

Uses: I use this when I move a service between servers or regions. I lower the TTL days ahead, move, watch the old address drain, and only then raise the TTL again.

Dimensions: At ten times the traffic the resolver cache is the load balancer, handing out addresses in turn. What breaks is a single nameserver, and I notice it when one resolver in the world stops answering while the rest work fine. It costs at least two nameservers on different networks and a habit of testing the failover. The trade is that DNS failover is slow by design, because the TTL decides the speed, not me.

Read the TTL. Every DNS answer carries a time to live, in seconds. It says how long you may keep the answer before asking again. I read it with dig, because the TTL decides how fast a change spreads. A TTL of 15 seconds means a change spreads in seconds. A TTL of a day means it spreads in a day.

$ dig +noall +answer example.com
example.com.            15      IN      A       198.18.48.185

Every flag in the commands above, letter by letter.

  • dig +noall: +noall turns off every section of dig's output. The question, the answer, the authority, all of it goes dark.
  • dig +answer: +answer turns the answer section back on. The pair together means show me only the answer. The second column in the answer is the TTL, 15 here. That number is how long the answer may be cached.
  • Without either flag, dig prints the question, the answer, and footer text with the query time and the server used. That is useful once. For a quick lookup, the two flags cut the noise.

Check it: run dig +noall +answer example.com, then plain dig example.com. The first prints one line. The second prints the question, the answer, and the footer.

Try it: run dig and read the number in the second column. It is 15. That means this answer may be kept for 15 seconds, and no longer. On a normal resolver, running it again a few seconds later shows a smaller number, because the cache is aging. When it hits zero, the next query asks the server again.

The name with no mail server. A domain can exist and still receive no mail. The MX record says where mail goes, and example.com answers with 0 ., which means nowhere. I check the MX record before I wonder why mail bounces, because the answer is often that there is no mail server at all.

$ dig +short MX example.com
0 .

Every flag in the commands above, letter by letter.

  • dig +short: +short prints only the answer value, nothing else. One line, no sections, no TTL.
  • MX is the record type, the mail servers. The answer here is 0 ., which means this domain takes no mail. A domain that takes mail prints a number and a server name instead.
  • One more flag worth knowing: dig -x 127.0.0.1 is a reverse lookup. It asks for the name behind an address, instead of the address behind a name. With +short it prints one line: localhost.

Check it: run dig +short MX example.com. It prints 0 . Then run dig -x 127.0.0.1 +short. It prints localhost.

Try it: run the command above, then run it for a domain you know receives mail. The second one returns a real server name. The difference is the whole answer to why one domain gets mail and the other does not.

Master — Running your own authoritative nameserver

Definition: At this level I run authoritative DNS as a service: anycast addresses so the nearest server answers, monitoring on every nameserver, DNSSEC signing so answers can be verified, and rate limits so a flood of questions cannot take the service down. I treat DNS as critical infrastructure, because when DNS is down, everything is down, even though every server is healthy.

Types:

  • Anycast, where many servers share one address and the network picks the nearest.
  • DNSSEC, where answers carry a signature.
  • Split-horizon DNS, where inside and outside the network get different answers.

Uses: I use this when a company depends on its names the way it depends on power: I monitor resolution from outside networks, not just from inside, and I test what happens when a nameserver dies.

Dimensions: At a hundred times the traffic the nameservers are spread across regions and the answers are tuned per region. What breaks is the thing nobody watches, usually a domain registration expiring, and I notice it from an outside check, not from any dashboard. It costs constant outside monitoring and renewal alerts. The trade is that DNS is invisible when it works and total when it fails, so it earns the same care as the database.

Stale answers. A DNS change does not reach everyone at once. Every resolver that cached the old answer keeps serving it until the TTL runs out. I plan a change around the TTL, not around the moment I make it. If the TTL is a day, I lower it to minutes a day before the change, then I make the change.

$ dig +noall +answer example.com
example.com.            15      IN      A       198.18.48.185

Try it: run dig and read the TTL number in the second column. That number is how long the old answer can live after you change it. A 15 second TTL means a fast cutover. An 86400 second TTL means a full day of mixed answers. Lower the TTL first, change second.

Go deeper

The resolver and the authoritative server are different jobs, and mixing them up causes real confusion. Your resolver asks questions for you. An authoritative server answers questions about names it owns. A server can do both jobs, but the jobs are different, and the debugging starts by asking which job failed.

Round robin DNS is the simplest load balancing there is. One name, several A records, and the resolver hands them out in turn. It is cheap and it works, until one of the addresses dies and the resolver keeps handing it out. DNS has no health check built in, so dead addresses keep getting served until someone removes the record.

DNS over UDP is one packet out and one back, which is why it is fast. Big answers do not fit in one UDP packet, so the resolver retries over TCP. That is normal. A DNS answer that always needs TCP is a sign the record set is large, often because of DNSSEC signatures.

Negative caching is the part people forget. When a name does not exist, the resolver remembers that too, for a short time. During that time, creating the name will not help anyone who already asked, because their resolver will keep answering that it does not exist until the negative TTL expires.

At 10x the traffic, the shape holds and the numbers grow. The resolver cache does more work, because more questions repeat. I watch the cache hit rate at the resolver the way I watch the hit rate at a web cache. A falling hit rate means new names, or TTLs that are too short.

At 100x, the design changes. The authoritative servers are anycast, so the nearest one answers. The TTLs are tuned per record, not set once for the whole zone. And the domain registration itself is monitored, because an expired domain takes down everything, and no amount of engineering fixes that in a hurry.

# ask the authoritative server directly (teaching example)
dig @ns1.example.com example.com A

# example answer section
# example.com.  300  IN  A  93.184.216.34

# ask for the mail servers instead
dig example.com MX

Every flag in the commands above, letter by letter.

  • dig @ns1.example.com: the @ picks which server to ask, instead of your default resolver. The name after @ is the server. Change it and you ask a different server the same question.
  • example.com A: A is the record type, the IPv4 address. dig example.com MX asks the same server for the mail servers instead.
  • One more flag worth knowing: dig +trace follows the delegation from the root down, one server at a time. It shows the walk instead of asking your resolver to do it. It needs direct access to the root servers, so on a network that forces a local resolver it times out.

Check it: run dig @198.19.0.1 example.com A +short, asking your local resolver by address. It prints 198.18.48.185. The @ is how you pick the server.

What the snippet means: dig asks a DNS question and shows the answer. The @ part picks which server to ask. The number 300 is the TTL in seconds. IN means internet class, A means the record type, and the address at the end is the answer. Asking the authoritative server directly skips every cache, so what it returns is the truth, not a memory.

Try it: read the TTL. Run dig +noall +answer example.com and read the number in the second column. It is 15. That number is the whole reason DNS changes take time to reach everyone. A change to this record spreads in 15 seconds. A record with a TTL of 86400 spreads in a day. The TTL is the propagation delay, written on the answer itself.

$ dig +noall +answer example.com
example.com.            15      IN      A       198.18.48.185

One time it broke

What broke

We moved a service to a new server on a Friday evening. The new server was up. The old server was still up. Half the users reached the new one and half reached the old one, for a whole day. Nothing was broken. Nothing was down. The move had worked, and the users were still split.

Timeline

  • 18:00,we changed the A record to the new address and confirmed it from our own machines.
  • 18:30,the first report came in: some users still saw the old version.
  • 19:00,I checked the authoritative server. It returned the new address. The change was live.
  • 19:20,I checked the TTL on the old record. It was 86400, a full day, set years ago and never reviewed.
  • 19:40,I explained to the team: resolvers that cached the old answer before the change may serve it for up to a day. There is no button to clear the world's caches.
  • Next day,the split closed on its own as TTLs expired. Every user reached the new server by the same time the next evening.

What changed after

Three things. First, every record we might move gets a short TTL, 300 seconds, as a standing rule. Second, before any move, we lower the TTL days ahead, so the old answer dies fast when we change it. Third, we raise the TTL back up after the move settles, because a short TTL on a stable record is just extra lookups forever. The lesson was not that DNS is slow. The lesson was that the TTL is a decision, and ours had been made by accident years earlier.

If you remember nothing else

Design exercise

Design the DNS setup for a shop that runs on two servers in two regions. Users must reach a working server if one region goes down. The shop moves servers about once a month.

Clarifying questions to ask

  • What TTL do you put on the shop's A records, and why?
  • One name, two A records. What happens when one server dies?
  • How do you prepare the TTL before a planned move to a new server?
  • What breaks if the domain registration expires, and how do you prevent it?
  • How do you test that the failover works, without taking the shop down?

Rough sizing

Start with the TTL math. A TTL of 300 seconds means a bad address is served for at most five minutes after you remove it, and a resolver asks again every five minutes, which is about 12 lookups an hour per resolver. A TTL of 86400 means one lookup a day per resolver, and a bad address served for up to a day. For a shop that moves monthly, 300 seconds on the A records costs a few extra lookups and buys a five-minute failover. Two A records split the traffic roughly in half, so each server needs to carry the full load alone when the other dies.

Trade-offs to name

Short TTLs buy fast failover and cost extra lookups. Two A records buy simple load sharing and cost the risk of serving a dead address until the TTL expires. DNS failover is slow by design, so if the shop needs failover in seconds, DNS alone is not the answer, and a load balancer in front of the servers is. And the registration itself is the single point of failure no TTL can fix, so renewal alerts go to more than one person.

Next in this series
Lesson F21 — TLDs: The End of Every Name