Platform Engineering · Lesson 148

Certificates and Keys: Who You Are, and Who Says So

Lesson 147 showed you the certificate arriving in the handshake. This lesson is where that document comes from. A private key, a request, a signature, and a chain: that is a certificate. You will make one four ways, with OpenSSL, with cfssl, with mkcert, and with Python, and you will know which type to buy and which key to pick. By the end you will have served a page over HTTPS from a certificate you made yourself, and you will have seen a browser refuse one it does not trust.

Difficulty: Advanced · Know this first: Lesson 147 - TLS from Scratch: The Handshake That Locks the Wire

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The certificate, mapped: key, request, signature, trust

Start with a secret no one else holds. Add one step at a time until a browser trusts the page.

7 steps to a trusted page4 ways to make one90 days of life
Key
1 min
CSR
1 min
CA
2 min
Certificate
1 min
Chain
1 min
Name
1 min
Trust refused
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the key, the signing, and the serving change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: a bigger key and a longer life cost time at signing, and more names cost time at serving.

2048

2048 is the safe default. 4096 is stronger and slower to make. ECDSA P-256 matches RSA 2048 strength with a 256 bit key, and it is not on this slider.

90

Ninety days is the modern life for a public certificate. Longer lives mean fewer renewals and a longer window when a stolen certificate stays valid.

3

Every name the service answers on goes in the list: the bare domain, www, and each subdomain. A missing name is a refused connection.

Try this: (1) Set key to 4096, life to 825, names to 10. The key is about 4 min, the signing about 9 min, the serving about 3 min, and the total is about 21 min. (2) Set key to 2048, life to 30, names to 1. The key is about 2 min, the signing about 1 min, the serving about 1 min, and the total is about 9 min. Same service. Only the key, the life, and the list changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is the one command from step 1, ready to edit. Change it, then run the check. The check looks for the nine parts a self-signed certificate command needs: the tool, the output type, the new key, the key file, the cert file, the life, the unencrypted key flag, the subject, and the names.

Try breaking it on purpose: delete the -addext part, or change -days 365 to -days 0. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a second DNS name to the SAN list.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The command has a tool, a type, a key, two files, a life, an unencrypted key, a subject, and a name list. Run it and two files appear: key.pem and cert.pem.

Basic to master

I take the same trip four times here. First I learn what a certificate is. Then I learn the types and the names. Then I learn the keys and the chains. Last I run it for the world.

Basic - A signed document that names a key

Definition: A certificate is a signed document. It says a name belongs to a public key, it names who signed it, and it names the dates it is good for. The private key stays with the owner. The public key travels in the certificate. A signature from an authority the browser trusts is what makes the document believable, and the chain of signatures is how the browser checks it.

Types:

  • A private key, which is the secret only the owner holds.
  • A public key, which travels in the certificate for anyone to see.
  • A certificate, which binds a name to a public key with a signature and dates.
  • A CA, which is the signer whose public half sits in every browser.

Uses: I use this for every site I serve and every API I call. Try it now: run the one command from the Try it box, then read the certificate back with openssl x509 -in cert.pem -noout -subject -issuer -dates. Then practice making certificates in the TLS labs.

openssl x509, flag by flag. This is the command that reads a certificate. I use it to check what I made before I hand it to a server.

  • -in names the certificate to read. Without it openssl reads from stdin, and a pasted certificate is easy to get wrong.
  • -noout prints no certificate. Without it the whole PEM prints after the fields I asked for, and the answer drowns in base64.
  • -text prints the whole decoded certificate. It is long. I pipe it to grep when I want one line, like the signature algorithm or the key size.

Flags you will meet next: -issuer prints who signed it, -text prints the whole decoded certificate, and -checkend answers whether it is still valid.

Try it: run it against a certificate you made, then change -days and make another. The -dates output should move by exactly the days you changed.

Dimensions: At low traffic everything is nearly free. What breaks first is the name: a certificate for the wrong name, or no SAN at all. It costs one careful read of the SAN list before you sign.

Try it: Basic

I read a certificate back with openssl x509. What three things does the document itself name?

Now do it: Run openssl x509 -in cert.pem -noout -subject -issuer -dates on a certificate you made. I ran it here and it printed the name, the signer, and the dates.

Intermediate - Types: DV, OV, EV, wildcard, SAN

Definition: DV, OV, and EV are three levels of checking before the signature. DV checks you control the domain, usually by answering a challenge on the domain. OV checks the company is real, in public records. EV checks the company hardest, with legal documents and a callback. A single-domain certificate names one name. A wildcard names one level under a domain, like *.example.com. A multi-domain certificate lists each name in the SAN. Wildcards are convenient. Lists are exact.

Types:

  • A DV certificate, which checks domain control only, and is the common case.
  • An OV certificate, which checks the company is real, and takes longer.
  • An EV certificate, which checks the company hardest, and takes longest.
  • A wildcard, which covers one level under a domain, and a SAN list, which names each name.

Uses: I use DV for most services, because the check is fast and the crypto is the same. I use OV or EV when the business needs the company check on paper. Try it now: list the names one of your services answers on, and say for each one whether a wildcard or a SAN list fits it better. Then practice the choice in the TLS labs.

Dimensions: As the service grows, the name list grows, and the wildcard starts to look easy. What breaks is the second level: api.v2.example.com is not covered by *.example.com. It costs a SAN list that names every level, or a second certificate.

Try it: Intermediate

My service answers on api.v2.example.com. Does my *.example.com wildcard cover it?

Now do it: List the names one of your services answers on, and say for each one whether a wildcard or a SAN list fits it better.

Advanced - RSA against ECDSA, chains, self-signed against CA-signed

Definition: RSA and ECDSA are two kinds of keys. RSA 2048 is the safe default, big and slow to make, understood everywhere. ECDSA with the P-256 curve gives the same strength with a 256 bit key, a smaller certificate, and a faster handshake, and it is the modern pick. A CA-signed certificate has a chain to a CA the browser trusts. A self-signed certificate signed itself, so it has no chain, and the browser refuses it unless you add the signer by hand. The lock is the same either way. The trust is the difference.

Types:

  • An RSA key, 2048 bits, the safe default, big and slow.
  • An ECDSA key, P-256, same strength, smaller and faster.
  • A CA-signed certificate, with a chain a browser can walk.
  • A self-signed certificate, with no chain, trusted only by hand.

Uses: I use RSA when the client is old and only speaks RSA. I use ECDSA for new services, because the handshake is faster and the certificate is smaller. I use self-signed only in the lab, never in public. Try it now: make the ECDSA version from Go deeper and compare its size with the RSA one. Then practice key choices in the TLS labs.

Dimensions: At ten times the services, the keys are the inventory. What breaks is the key nobody can find, or the private key that leaked into version control. It costs a key store, a rotation schedule, and a rule: private keys never go in version control.

Try it: Advanced

I need a key for a new service and the clients are modern. Which key do I pick?

Now do it: Make the ECDSA version from Go deeper and compare its size with the RSA one. On my machine the ECDSA cert was 583 bytes and the RSA cert was 1119 bytes.

Master - Keys at scale in a real system

Definition: At this level the certificate stops being a file on one server and becomes a pipeline product. The pipeline makes the key, signs the request, installs the certificate, and renews it before it ends. The CA is private for inside traffic and public for outside traffic, and the two never mix. Keys live in a store, not in files on disk, and rotation is rehearsed, so a leaked key is replaced in minutes, not days. And I keep a list of every certificate, where it lives, and when it ends, because the one I forgot is the one that expires.

Types:

  • A private CA, which signs inside traffic that never leaves the cluster.
  • A public CA, which signs outside traffic that browsers must trust.
  • A key store, which holds private keys where files cannot leak them.
  • A renewal pipeline, which replaces a certificate before it ends, without a human.

Uses: I use this when more than one team serves through the same edge, or when an expired certificate can take down a service other teams depend on. It is also how I review a setup I did not build: I ask where the keys live, who can sign, and what happens when a certificate ends. Try it now: list every certificate on a service you own, with its end date. If you cannot list them, that is the first gap to close.

Dimensions: At a hundred times the certificates, the expiry date is not the bottleneck, the blast radius is. What breaks is the single CA every service depends on, or the renewal that only one person knows how to run. It costs automation, a second CA, and a rehearsed rotation.

The hand-run renewal pipeline here has a managed counterpart: the cloud requests the certificate, proves the domain in DNS, and renews it before it ends. That path is spelled out in Lesson 149.

Try it: Master

A key leaks on a service I own. What should already be true so I can replace it in minutes?

Now do it: List every certificate on a service you own, with its end date. If you cannot list them, that is the first gap to close.

Go deeper

Every flag of the one command, in order. Do this: run openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1", then read this list against the command you just ran. Expected output: two files, key.pem and cert.pem, and the dates one year apart when you read the certificate back.

openssl req with a SAN, flag by flag. This is the form I actually use. The SAN list is what browsers check, and a certificate without it is rejected even when the name matches.

  • -x509 signs the certificate with its own key instead of sending a request to a CA. Without it req prints a request and stops, and there is no certificate to use.
  • -newkey rsa:2048 makes a fresh RSA key with 2048 bits. Without it you must pass an existing key with -key, and the command has nothing to sign with.
  • -keyout names the file for the private key. Without it the key prints to stdout, and it is easy to lose it or paste it somewhere it does not belong.
  • -out names the file for the certificate. Without it the certificate prints to stdout instead of landing in a file the server can read.
  • -days sets how many days the certificate stays valid. Without it the default is 30 days, and a lab certificate made today is dead before the month is out.
  • -nodes writes the private key with no passphrase on it. Without it openssl stops and asks for a passphrase, and every later use of the key stops to ask again. I use -nodes only for lab keys. A real key I ship never has -nodes on it.
  • -subj "/CN=localhost" sets the subject without the interactive prompts. Without it openssl asks for each field one by one: country, state, name. The /CN= part is the common name, the name the certificate is for.
  • -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" adds an extension to the certificate. subjectAltName lists the names and IPs the certificate is valid for. Without it the certificate has no SAN, and modern clients reject it even when the CN matches, because they check the SAN, not the CN.

Flags you will meet next: -newkey ec with -pkeyopt ec_paramgen_curve:P-256 makes the EC form of the same certificate.

Try it: run it, then read the SAN back with openssl x509 -in cert.pem -noout -ext subjectAltName. The DNS name and the IP should both be there.

openssl req   the tool that makes requests and certificates
-x509           output a certificate, not a request
-newkey rsa:2048  make a new RSA key, 2048 bits
-keyout key.pem   write the private key to key.pem
-out cert.pem     write the certificate to cert.pem
-days 365        the certificate is good for 365 days
-nodes           do not encrypt the private key (in OpenSSL 3.0 the modern spelling is -noenc)
-subj "/CN=localhost"  the subject, set on the command line, no prompts
-addext "subjectAltName=..."  add the SAN names the browsers actually read

What the list means: each flag does one job, and leaving one out changes the result. Without -x509 you get a request, not a certificate. Without -nodes the key is encrypted and the server asks for a password on every start. Without -subj the command stops and asks you questions. Without -addext the certificate has no SAN and browsers refuse it.

The ECDSA version, one command. Do this: run openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout ec-key.pem -out ec-cert.pem -days 365 -nodes -subj "/CN=localhost". Expected output: the same dots and dashes, and two files. Then run openssl x509 -in ec-cert.pem -noout -text | grep -E 'Signature Algorithm|Public-Key|NIST CURVE'. Expected output: ecdsa-with-SHA256, Public-Key: (256 bit), and NIST CURVE: P-256. Compare the file sizes with ls -l ec-cert.pem cert.pem. Expected output: the ECDSA certificate is smaller than the RSA one.

ls -l, flag by flag. One flag, and it turns a list of names into a list of facts.

  • -l prints the long form: permissions, owner, size, and date. Without it ls prints bare names, and I cannot tell if a file is readable or how big it is.

Flags you will meet next: -h prints sizes in KB and MB instead of bytes, and -a shows the hidden files too.

Try it: run ls and then ls -l on the same folder. The second one shows who owns each file, how big it is, and when it changed. The first one shows names only.

openssl req with an EC key, flag by flag. Same command, different key. The EC form is shorter to write and faster to use, and the curve flag is the part people forget.

  • -x509 signs the certificate with its own key instead of sending a request to a CA. Without it req prints a request and stops, and there is no certificate to use.
  • -newkey ec makes a fresh EC key instead of RSA. The curve comes from -pkeyopt. Without the curve the key has no shape, and the command fails.
  • -pkeyopt ec_paramgen_curve:P-256 picks the P-256 curve for the EC key. Without it openssl does not know which curve to use, and the key generation fails.
  • -keyout names the file for the private key. Without it the key prints to stdout, and it is easy to lose it or paste it somewhere it does not belong.
  • -out names the file for the certificate. Without it the certificate prints to stdout instead of landing in a file the server can read.
  • -days sets how many days the certificate stays valid. Without it the default is 30 days, and a lab certificate made today is dead before the month is out.
  • -nodes writes the private key with no passphrase on it. Without it openssl stops and asks for a passphrase, and every later use of the key stops to ask again. I use -nodes only for lab keys. A real key I ship never has -nodes on it.
  • -subj "/CN=localhost" sets the subject without the interactive prompts. Without it openssl asks for each field one by one: country, state, name. The /CN= part is the common name, the name the certificate is for.

Flags you will meet next: -addext adds the SAN list, and -newkey rsa:4096 makes a bigger RSA key when the policy asks for one.

Try it: run it, then read the key type back with openssl x509 -in ec-cert.pem -noout -text | grep Public-Key. It should say NIST P-256, not RSA.

The second way: cfssl, for teams that sign many certificates from config files. Do this: write a ca-config.json that names a server profile with a 90 day life, a ca-csr.json that names your CA, and a server-csr.json that names app.local with its hosts list. Then run cfssl gencert -initca ca-csr.json | cfssljson -bare ca. Expected output: ca.pem, ca-key.pem, and ca.csr. Then run cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare server. Expected output: server.pem, server-key.pem, and server.csr. The flags: -initca makes a new CA, -ca and -ca-key name the signer, -config names the profiles file, -profile picks the server profile, and cfssljson -bare writes the files without wrapping them in JSON.

cfssljson -bare, flag by flag. cfssl prints JSON. cfssljson turns it into files. -bare is the flag that makes the files usable.

  • -bare server writes plain files named server.pem, server-key.pem, and server.csr. Without -bare the output is wrapped in the API response format, and openssl cannot read it until I unwrap it. The word after -bare is the file prefix, not a flag.
  • | the pipe sends cfssl output straight into cfssljson. Without it I would save the JSON to a file and run a second command on it.

Flags you will meet next: -f file reads JSON from a file instead of stdin, and -stdout prints the response instead of saving it.

Try it: run the gencert line with -bare and then without it. With -bare you get server.pem on disk. Without it you get a JSON document on stdout, and openssl cannot read it.

cfssl gencert, flag by flag. cfssl signs certificates from JSON files. The flags say which CA signs, which profile applies, and where the request lives.

  • -initca makes a new CA from the CSR file. Without it gencert signs with an existing CA, and it needs -ca and -ca-key instead.
  • -ca=ca.pem names the CA certificate to sign with. The new certificate chains back to this one.
  • -ca-key=ca-key.pem names the CA private key. The signature on the new certificate comes from this key. It never leaves the machine that holds the CA.
  • -config=ca-config.json names the signing config file, where the profiles live. Without it cfssl uses its defaults, and the profile I want does not exist.
  • -profile=server names the profile inside the config. The profile sets the expiry and the usages, like server auth. Without it the default profile signs, and the certificate may not be valid for a server at all.
  • server-csr.json the request file, named last. Use - for stdin. The CN and the hosts list inside it become the certificate name and its SANs.

Flags you will meet next: -hostname overrides the names in the CSR, and -remote asks a cfssl server to sign instead of signing locally.

Try it: run the -initca line first, then the server line. The first makes ca.pem and ca-key.pem. The second makes server.pem. Verify the chain with openssl verify -CAfile ca.pem server.pem, and expect OK.

The third way: mkcert, for a laptop. Do this: run mkcert -install once, which makes a local CA and puts it in the system trust store, then run mkcert app.local localhost 127.0.0.1 ::1. Expected output: a list of the names, the certificate path app.local+3.pem, the key path app.local+3-key.pem, and an expiry date a little over two years out. The browser trusts the result at once, because the CA is in the trust store. If the browser step fails on a read-only store, the certificate still verifies with openssl against the mkcert root, and the system store install is the part that matters.

mkcert, flag by flag. mkcert makes a local CA and signs certificates for local names. It is the fastest honest way to get a trusted certificate on a laptop.

  • -install puts the local CA into the system trust store. Without it the browser does not trust the certificates mkcert makes, and every page shows a warning.
  • app.local localhost 127.0.0.1 ::1 each name becomes a SAN on one certificate. app.local and localhost are names. 127.0.0.1 and ::1 are IPs. Without the IP entries, a browser that calls the IP gets a name mismatch, even though the certificate is trusted.

Flags you will meet next: -cert-file and -key-file name the output files, -ecdsa makes an EC key, and -client makes a client certificate.

Try it: run mkcert -install, then make the certificate, then open the name in a browser. With -install the page loads with no warning. Without it the same certificate shows a warning, because the CA is not trusted yet.

The fourth way: Python, for code that makes its own certificates. Do this: save the script from the TLS labs, which builds a CA and a leaf with the cryptography library, and run it. Expected output: four files, py-ca.pem, py-ca-key.pem, py-leaf.pem, and py-leaf-key.pem, and the line wrote py-ca.pem py-ca-key.pem py-leaf.pem py-leaf-key.pem. Then run openssl verify -CAfile py-ca.pem py-leaf.pem. Expected output: py-leaf.pem: OK. The same chain, the same check, built by code instead of by a command.

openssl verify, flag by flag. One job: check a certificate against a CA. The answer is one word, and I trust it more than my eyes on the dates.

  • -CAfile names the CA file to trust. Without it the check uses the system store, and a private CA fails even when it signed the certificate correctly.
  • cert.pem the certificate to check, named last. The answer is one word: OK, or the reason it failed.

Flags you will meet next: -partial_chain accepts a chain that ends at an intermediate, and -untrusted adds extra chain files to the check.

Try it: run it with the right CA file, then with no CA file at all. The first prints OK. The second fails, even though the certificate did not change. The trust store is the variable, not the certificate.

Serve the page from your own certificate. Do this: save this script as serve.py in the folder with leaf.pem and leaf-key.pem, then run it, then run curl --cacert ca.pem https://localhost:8443/index.html in another shell.

curl --cacert, flag by flag. A private CA is not in the system store. This flag tells curl where the trust lives, for this one call only.

  • --cacert ca.pem names the CA file to trust for this one call. Without it curl uses the system store. A private CA is not in that store, so the check fails even when the certificate is fine.

Flags you will meet next: -k skips verification entirely. It makes the error go away and the trust go away with it. I never use it outside a lab.

Try it: run it with --cacert and the right CA file, then without it. With it the request works. Without it curl fails with a certificate error, even though the server certificate is fine.

import http.server, ssl
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain("leaf.pem", "leaf-key.pem")
srv = http.server.HTTPServer(("127.0.0.1", 8443), http.server.SimpleHTTPRequestHandler)
srv.socket = ctx.wrap_socket(srv.socket, server_side=True)
srv.serve_forever()

What the script means: an SSL context for a server, the leaf certificate and its key loaded into it, a plain HTTP server on port 8443, and the socket wrapped so every byte travels inside TLS. Expected output from curl: hello from the tls lab. Expected output from curl without --cacert: exit code 60, the certificate could not be verified. Practice all four ways in the TLS labs.

At 10x the services, the shape holds and the numbers grow. At 100x, the design changes: the pipeline makes the keys, a private CA signs the inside, a public CA signs the outside, and renewal runs on its own. The certificate never stops being a key, a request, a signature, and a name list. It just gets more readers. Lesson 149 is where each one gets checked.

One time it broke

What broke

A team shipped a certificate with only a CN and no SAN. The certificate was valid, the chain was fine, and the dates were good. Every modern browser refused the page anyway, because browsers stopped reading the CN years ago and only read the SAN list. The page was down for everyone with a new browser and fine for no one, and the team spent a morning blaming the server before someone read the certificate.

Timeline

  • 09:12,the new certificate is installed and the server is reloaded. The deploy log is clean.
  • 09:15,the first user reports the page will not load. The browser says the certificate is not valid for the name.
  • 09:30,I run openssl x509 -in leaf.pem -noout -ext subjectAltName and get back: No extensions in certificate.
  • 09:35,I read the old certificate the same way. It has a SAN list with four names. The new one has none.
  • 09:50,I make a new CSR with the SAN list, sign it, and install the new certificate.
  • 10:05,the page loads in every browser. The server config never changed. Only the certificate did.

What changed after

Two things. First, the signing checklist now has one line at the top: read the SAN list before you install. Second, the CSR template carries the SAN list, so a certificate cannot be made without it. Try it now: run openssl x509 -in on any certificate you own and read the SAN list. If it prints No extensions in certificate, that certificate is living on borrowed time.

If you remember nothing else

Build it yourself

Build your own lab CA, sign a leaf certificate with four SAN names, and serve a page over HTTPS from it. It passes when curl answers hello with --cacert and refuses without it, and when openssl verify says OK for the leaf and fails for a self-signed certificate.

Build it, step by step

  • Make a lab CA with openssl req -x509 and read it back: subject and issuer must match, which is how you spot a root.
  • Make a leaf CSR for app.local, write a leaf.ext file with four SAN names, and sign it with the lab CA for 90 days.
  • Read the leaf back with openssl x509 and confirm the issuer is the lab CA, the dates are 90 days apart, and the SAN list has all four names.
  • Run openssl verify -CAfile ca.pem leaf.pem until it says OK, then run it against the self-signed cert.pem from step 1 and read the error 18.
  • Start the Python HTTPS server from Go deeper and curl it three ways: with --cacert, with no flags, and with -k. Write down the output and the exit code for each of the three.

What success looks like

Success looks like this: leaf.pem says OK against the lab CA. The self-signed certificate fails with error 18. Curl with --cacert prints hello from the tls lab. Curl with no flags prints nothing and exits 60. Curl with -k prints hello and exits 0, and you can say why the -k one is not a fix.

If it fails, check this

If verify says unable to get local issuer certificate, you pointed -CAfile at the wrong file, or the leaf was signed by a different CA. If curl with --cacert still fails, the name in the address bar is not in the SAN list, so check the SAN, not the server. If the server will not start, the key and the certificate do not match: compare their moduli with openssl x509 -noout -modulus and openssl rsa -noout -modulus, which must print the same value.

Next in this series
Lesson 149 - Validating Certificates: Read One, Trust One, Catch a Bad One