Platform Engineering · Lesson 206

TLS in Production: Termination, mTLS, and Rotation

Lesson 149 taught you to check a certificate. This lesson is what happens after the check passes. A request hits the edge, the lock ends there, the traffic inside runs on its own rules, and the certificates get replaced while the traffic keeps flowing. I run this the way I run it on a real service. Termination at the edge. mTLS between services. Rotation without downtime. HSTS and OCSP stapling on top. By the end you will have made a client certificate, proved a server refuses a caller without one, and swapped a leaf certificate while the traffic kept flowing.

Difficulty: Master · Know this first: Lesson 149 - Validating Certificates: Read One, Trust One, Catch a Bad One

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: termination is cheap, fresh handshakes are the tax, and a refused client proof ends the trip early. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

Production TLS, mapped: edge, mTLS, rotation, answer

Start with a browser and an edge. Add one step at a time until the answer comes back, or the client proof refuses it.

7 stops on the trip2 certificates in the call1 swap, no downtime
Browser
0 ms
Edge
15 ms
Handshake
25 ms
Inside
5 ms
mTLS call
20 ms
Rotation
10 ms
Refused
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the edge handshake, the mTLS calls, and the rotation check change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the number of services, the number of internal calls, and the session reuse decide how slow the lock feels.

10

How many services sit behind the edge. Every service needs a certificate, and every certificate needs a rotation. More services mean more rotation work.

3

How many service-to-service calls one request makes. Every fresh call pays an mTLS handshake. A request that fans out pays it many times.

50

Share of internal calls that resume an old mTLS session instead of a fresh handshake. A resumed session costs about 1 ms. A fresh one costs about 10 ms.

Try this: (1) Set services to 50, calls to 8, reuse to 0. The edge is about 25 ms, mTLS about 80 ms, rotation about 25 ms, and the total is about 195 ms. (2) Set services to 1, calls to 1, reuse to 100. The edge is about 25 ms, mTLS about 1 ms, rotation about 1 ms, and the total is about 92 ms. Same request. Only the services, the calls, and the reuse changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is the one command that proves mTLS from the client side. Change it, then run the check. The check looks for the five parts an mTLS curl call needs: curl, the CA to trust, the client certificate, the client key, and a locked address. Start the mTLS lab server on port 8443 first, or the command has nothing to talk to.

Try breaking it on purpose: delete the --cert part, or change https to http. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding -s to silence the progress meter.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The command proves the client to the server and prints hello from the mtls lab. Run it without --cert and the server closes the connection: nothing printed, exit code 56.

Basic to master

I take the same trip four times here. First I learn where the lock ends. Then I learn the client proof. Then I learn to rotate and to set the headers. Last I run it for the world.

Basic - Termination: where the lock ends

Definition: Termination is where the public TLS connection ends. The edge holds the public certificate, runs the handshake, and forwards the request inside as plain HTTP. The services behind the edge never see the public lock. The browser talks to the edge, and the edge talks to the services. One public certificate serves the whole site.

Types:

  • Termination, where the public lock ends and the inside traffic begins.
  • The edge, which holds the public certificate and runs the handshake.
  • Plain HTTP inside, which is safe when the inside network is private and the edge is the only door in.
  • Keep-alive, which reuses one edge connection for many requests so the handshake is paid once.

Uses: I use this for every site I run. Try it now: start the lab backend and the Python terminator, run curl against the TLS port, and read the plain answer that comes back. Then run plain HTTP against the TLS port and watch it get no answer. Practice termination in the TLS labs.

Dimensions: At low traffic everything is nearly free. What breaks first is the door: a service reachable on plain HTTP from outside the private network. It costs one rule: the edge is the only door in.

Intermediate - mTLS: the client proves itself

Definition: mTLS is TLS with the client proving itself too. The server asks for a client certificate, checks it against the CA, and checks the name on it. A caller with no certificate gets no answer. A caller with a certificate from another CA gets no answer either. The service mesh lesson, the Linkerd lesson, and the Istio lesson all name mTLS, and this lesson is the first one that runs it. The identity lives in the certificate, not in the app.

Types:

  • A client certificate, which names the calling service.
  • CERT_REQUIRED, the server setting that demands a client certificate.
  • Alert 116, certificate required, which is the server refusing a caller with no certificate.
  • A CA both sides trust, which is what makes the check mean anything.

Uses: I use this whenever two services talk on a shared network, or when a call must be allowed only for named callers. Try it now: run the mTLS curl call with the client certificate, then without it, then with a certificate from another CA. Write down the three exit codes. Then practice mTLS in the TLS labs.

Dimensions: As the services grow, the handshakes grow, because every fresh call pays one. What breaks is the client certificate nobody watches: it ends, the calls fail, and the app logs say nothing. It costs a -checkend alert on client certificates too, not just server ones.

Advanced - Rotation, HSTS, and OCSP stapling

Definition: Rotation replaces a certificate before it ends, while both the old and the new verify OK against the same CA. HSTS tells the browser to refuse plain HTTP for the site for a year. OCSP stapling puts the revocation answer in the handshake, so the client never has to ask the CA. A missing staple is not a failure, just a slower check. I checked the live site for this lesson: it sends strict-transport-security: max-age=31536000, and it sends no stapled OCSP response, so its clients still ask the CA themselves.

Types:

  • Rotation, the swap of a leaf while both leaves verify OK.
  • The overlap window, which is the downtime-free time when both leaves are trusted.
  • HSTS, which tells the browser to refuse plain HTTP for the site.
  • OCSP stapling, which puts the revocation answer in the handshake.

Uses: I use this when a certificate ends on a service other teams depend on. Try it now: run openssl verify against both lab leaves, swap them, restart the server, and run the mTLS curl call before and after. Then run s_client with -status against the live site and read the OCSP response line. Then practice rotation in the TLS labs.

Dimensions: At ten times the certificates, the rotation is the inventory. What breaks is the client certificate, the intermediate, or the CA itself ending while everyone watches the leaf. It costs one list of every certificate, server and client, with a date next to each.

Master - TLS in production at scale

Definition: At this level TLS is a system, not a certificate. The edge fleet terminates for every public name. The inside traffic runs mTLS with short-lived certificates that rotate on their own. The CA is not one machine: there is a second CA, and a rehearsed switch to it. The expiry dates are watched in the monitoring system, with alerts at 30, 14, and 7 days, and an alert for the check itself failing. The version policy is enforced at the edge and checked in the logs. This is the last lesson of the track, and the last lesson of the course: Lessons 147, 148, and 149 taught the handshake, the certificate, and the check, and this lesson put them to work.

Types:

  • An edge fleet, which terminates TLS for every public name.
  • Short-lived mTLS certificates, which rotate on their own inside the cluster.
  • A second CA, with a rehearsed switch when the first one fails.
  • Watched dates, with alerts at 30, 14, and 7 days, and an alert for the check itself.

Uses: I use this when more than one team depends on the same certificates, or when an expired certificate can take down a service I do not own. It is also how I review a setup I did not build: I ask where TLS ends, who the client certificates belong to, what happens when the CA renews, and what fires first, the alert or the outage. Try it now: list every certificate you own, server and client, with its end date next to it.

Dimensions: At a hundred times the certificates, the expiry date is not the bottleneck, the blast radius is. What breaks is the single CA every service depends on, or the rotation that only one person knows how to run. It costs automation, a second CA, and a rehearsed rotation, and a reader who finished this lesson has the whole track to build it from.

Go deeper

The termination config, two ways. The first is the server config readers will meet most often. It was not run in this lab, because the lab machine has no web server installed: it is shown here so the reader can paste it on a server that has one, and the Python terminator below is the part that was actually run. Do this on a server with a web server installed: paste this block as the server for the site, point the certificate paths at your own files, and reload. Expected output: the site serves over HTTPS, and the backend still answers plain HTTP inside.

server {
  listen 443 ssl;
  server_name app.local;
  ssl_certificate /etc/ssl/certs/leaf.pem;
  ssl_certificate_key /etc/ssl/private/leaf-key.pem;
  ssl_protocols TLSv1.2 TLSv1.3;
  add_header Strict-Transport-Security "max-age=31536000" always;
  location / {
    proxy_pass http://127.0.0.1:8080;
  }
}

What the block means, line by line: listen 443 ssl says this server ends TLS on port 443. The two ssl_certificate lines point at the leaf and its key. ssl_protocols turns off everything below TLS 1.2. The add_header line sets HSTS on every answer. proxy_pass forwards the request to the plain backend on port 8080. The public lock ends here. The backend never sees it.

The terminator that was actually run. Do this: start the plain backend on port 8080 and the Python terminator on port 8444, then run curl -s --cacert ca.pem https://localhost:8444/. Expected output: hello from the plain backend. The curl call is locked. The backend answered plain HTTP. That gap is termination, running in front of you. Then run curl -s http://127.0.0.1:8444/. Expected output: nothing, and exit code 56. Plain HTTP to a TLS port gets no answer at all.

HSTS, read from a live site. Do this: run curl -sI https://mohankrishnapodile.com/ | grep -i strict-transport-security. Expected output: strict-transport-security: max-age=31536000. That number is seconds, and 31,536,000 seconds is one year. The header is set at the edge, so every answer carries it, and the browser remembers it after the first visit. A site with no such line is locked but not strict, and a user who types http still sends one plain request first.

OCSP stapling, read from the same live site. Do this: run echo | openssl s_client -connect mohankrishnapodile.com:443 -servername mohankrishnapodile.com -status 2>/dev/null | grep -i 'OCSP response'. Expected output, checked for this lesson: OCSP response: no response sent. The live site does not staple. Its clients still ask the CA themselves when they check revocation, which is slower but not broken. A site that staples prints a response block here instead. The -status flag is the whole check: it asks the server to staple the answer, and the answer tells you whether it did.

The rotation runbook, all in one place. Do this: run openssl verify -CAfile ca.pem leaf.pem leaf2.pem. Expected output: leaf.pem: OK and leaf2.pem: OK. Then run openssl x509 -in leaf.pem -noout -fingerprint and the same for leaf2.pem. Expected output: two different fingerprints. Same CA, same name, different keys, both trusted. Swap the files, restart the server, and run the mTLS curl call. Expected output: hello from the mtls lab, before the swap and after it. The fingerprints differ. The answer does not. That is the rotation.

At 10x the certificates, the shape holds and the numbers grow. At 100x, the design changes: the edge is a fleet, the inside certificates are short-lived and rotate on their own, and there is a second CA with a rehearsed switch. The mTLS check never stops being a client certificate against a CA. It just stops being a human running curl. The mesh lessons are where that fleet lives: the service mesh lesson, the Linkerd lesson, and the Istio lesson name mTLS as a feature, and this lesson is the teaching behind the name. The track is complete: Lesson 147 taught the handshake, Lesson 148 taught the certificate, Lesson 149 taught the check, and this lesson put them to work.

One time it broke

What broke

An internal service stopped answering, and nothing in the app logs said why. The calls failed, the dashboards showed the service as healthy, and the app had no error to show, because the failure happened before the app was called. The client certificate had ended the night before. The server asked for it, checked the dates, and refused the call. The service was fine. Its proof of identity had expired.

Timeline

  • 22:00,the client certificate ends. The next call after that hour is refused at the handshake.
  • 09:10,the first team notices the calls failing. The app logs are empty, so they check the service logs next.
  • 09:25,the server TLS log shows alert 116, certificate required, and a failed handshake for every call.
  • 09:40,the client certificate is read with x509 -dates. It ended at 22:00 the night before.
  • 10:05,a new client certificate is made from the same CA, and the calls work again.
  • 10:30,the team adds client certificates to the watched list, with the same 30, 14, and 7 day alerts the server certificates already had.

What changed after

Two things. First, client certificates are watched like server certificates, with alerts at 30, 14, and 7 days, because a client certificate ends just as quietly as a server one. Second, the runbook for a silent internal failure starts at the TLS log, not the app log, because a handshake failure never reaches the app. Try it now: run x509 -dates on your client certificate and write down its end date. If you cannot find your client certificate, that is the gap to close.

If you remember nothing else

Build it yourself

Build the mTLS pair and rotate the leaf without dropping traffic. Make a client certificate from the lab CA, start the mTLS server, prove the three curl cases, then make a second leaf, swap it in, and prove the traffic still flows. It passes when all three curl cases print what this lesson says they print, and the leaf swap changes the fingerprint without changing the answer.

Build it, step by step

  • Make a client certificate from the lab CA with openssl req and openssl x509 -req, and verify it with openssl verify -CAfile ca.pem client.pem.
  • Start the mTLS server on port 8443, run curl with --cert and --key, and write down the answer and the exit code.
  • Run the same curl call without --cert, and write down the exit code. Then run it with a client certificate from a different CA, and write down that exit code too.
  • Make a second leaf from the same CA, verify both leaves with one openssl verify command, and write down the two fingerprints.
  • Swap the leaf files, restart the server, run the mTLS curl call again, and write down the answer. Say in one line what changed and what did not.

What success looks like

Success looks like this: the client certificate verifies OK. Curl with the client certificate prints hello from the mtls lab and exits 0. Curl without it prints nothing and exits 56. Curl with the wrong CA certificate prints nothing and exits 56. Both leaves verify OK, the fingerprints differ, and the curl call prints hello before the swap and after it.

If it fails, check this

If curl with the client certificate still exits 56, check the server is the mTLS one: a plain server on the same port answers plain HTTP and the handshake fails differently. If openssl verify says unable to get local issuer certificate, the -CAfile is wrong, or the leaf was signed by a different CA. If the swap breaks traffic, the old leaf files are still in the lab as leaf-old.pem: copy them back, restart, and check the new leaf verifies before you swap again.

This series is complete
Lesson 01 - How a Request Travels

The cert checks out and the connection holds. Next lesson I walk through the attacks that show up anyway, and the fence that stops them, layer by layer.