Platform Engineering · Lesson 146

Docker in Production: Data, Secrets, Limits, and Security

Lessons 140 to 145 built images and wired apps. This lesson is the one where the app runs for real users, and the mistakes cost more than a slow build. Data lives outside the container. Secrets never land in an image. Every container has a limit. Every container proves it is healthy. Nothing runs as root, and every image is scanned. By the end you can look at a production setup and name what is missing.

Difficulty: Master · Know this first: Lesson 144 — Docker Compose: A Whole App in One File

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

Production, mapped: data, secrets, limits, scan

Start with data outside the container. Add one control at a time until the app is limited, sealed, non-root, and scanned.

6 controls1 leaked secret1 starved host
Volume
2 min
Bind mount
1 min
Tmpfs
1 min
Secret
2 min
Limit
6 min
Health gate
2 min
Root + flaw
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the memory pressure, the CPU pressure, and the scan load change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: more containers with smaller limits mean more pressure, and the pressure is what makes a host crawl.

6

How many containers share one host. Every container adds memory pressure, CPU pressure, and one more image to scan.

512

How much memory each container may use. A small limit means more pressure per container. A big limit means fewer containers fit on the host.

4

How much CPU each container may use, in half-cores. 2 is one core, 4 is two cores. A small limit means the app slows under load.

Try this: (1) Set containers to 20, memory limit to 128, CPU limit to 1. The memory is about 80 min, the CPU about 40 min, the scan about 10 min, and the total is about 132 min. (2) Set containers to 1, memory limit to 4096, CPU limit to 8. The memory is about 0 min, the CPU about 0 min, the scan about 1 min, and the total is about 3 min. Same host. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is the production compose override from this lesson, ready to edit. It caps the web service, runs it as a non-root user, keeps it read only, mounts its secret as a file, and checks its health on a schedule. Change it, then run the check.

Try breaking it on purpose: delete the secrets block from the web service. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like raising the memory limit to 1G.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The override has deploy limits, secrets, a read only filesystem, a non-root user, and a healthcheck. This service would run safe in production.

Basic to master

I take the same setup four times here. First I learn the six controls and what each one does. Then I learn the traps: a secret in an env var, a container with no limit. Then I learn to prove each control works. Last I run a setup I can defend.

Basic — Six controls, six jobs

Definition: A volume holds data outside the container. A bind mount maps a host folder in, for dev. A tmpfs mount lives only in memory, for secrets. A secret comes from a secret store and is mounted as a file. A resource limit caps what one container may use. A healthcheck proves the app can answer. A scan gate blocks an image with a known critical flaw.

Types:

  • A volume, which holds the data that must survive the container.
  • A secret, which is mounted as a file and never shows its value.
  • A memory limit, which kills the container when it is hit.
  • A CPU limit, which slows the container instead of killing it.

Uses: I use this to run one app safely: a volume for the data, a secret for the password, a limit on the container, and a healthcheck on the app. Try it now: delete the secrets block in the Try it box and run the check to see which control the file lost. Then practice each control in the Docker labs.

Dimensions: At low traffic everything is nearly free. What breaks first is the missing control: a password in an env var, a container with no limit, a healthcheck that was never added. It costs one review of the file against the six controls.

Intermediate — The traps: leaks and starvation

Definition: A secret in an env var leaks through inspect output, through logs when the app prints its environment, and through image history when it is baked in with ENV. A container with no memory limit grows until the host runs out, and the other containers slow to a crawl. A healthcheck that always passes is not a check. Each trap has one fix: a secret file, a limit on every container, and a check tested by breaking the endpoint on purpose.

Types:

  • A leaked secret, which is visible in logs, inspect, or image history.
  • A starved host, which is one container with no limit eating the memory.
  • A check that never fails, which proves nothing.
  • A root user, which the scan flags and the deploy blocks.

Uses: I use this when a production setup misbehaves. I check the secret handling, the limits, and the healthcheck, in that order. Try it now: change the memory limit in the Try it box to 128M and think through what happens to a busy app. Then put it back and run the check. Then practice the traps in the Docker labs.

Dimensions: As the team grows, the traps multiply. What breaks is the setup nobody reviewed: a secret in the file, a container with no limit, a scan gate that never blocks. It costs a review checklist, not more rules.

Advanced — Prove each control works

Definition: A control you have not tested is a hope. Prove the volume: remove the container and read the data back. Prove the secret: run inspect and confirm no password shows. Prove the limit: run docker stats under load and watch the usage stop at the cap. Prove the healthcheck: break the endpoint and watch the status flip. Prove the scan gate: push an image with a known flaw and watch the deploy block. Each proof is one command, and each one takes a minute.

docker stats --no-stream, flag by flag. stats shows live CPU and memory per container. --no-stream turns the live view into one snapshot I can read and pipe.

  • --no-stream prints one snapshot and exits. Without it the table keeps refreshing until I stop it, and I cannot pipe the output anywhere.

Flags you will meet next: --format '{{.Name}} {{.CPUPerc}} {{.MemUsage}}' prints only the columns you name.

Try it: run it with --no-stream and without it. With it the command prints once and exits. Without it the table keeps refreshing until you stop it.

Types:

  • A volume proof, which is data surviving a container removal.
  • A secret proof, which is inspect showing the name and never the value.
  • A limit proof, which is docker stats stopping at the cap.
  • A scan proof, which is a blocked deploy on a known flaw.

Uses: I use this before any production deploy. I run the five proofs, and I ship only when all five pass. Try it now: change the user in the Try it box to root and run the check. The check fails, because the file must run as a non-root user. That is the proof, in miniature. Then practice the proofs in the Docker labs.

Dimensions: At ten times the containers, the proofs are the difference between a setup that is safe and a setup that looks safe. What breaks is the proof nobody ran: the volume that was never tested, the limit that was never hit, the gate that never blocked. It costs five minutes before every deploy.

Read the memory numbers before the OOM killer does. A container that dies with exit code 137 was killed for memory, and the fix is not to restart it. I read the memory usage from the stats and compare it with the limit. If the usage climbs steadily and never comes down, the app leaks. If it spikes once, the limit is too small for the spike. The numbers decide which fix is right, and I read them before I change anything.

  • Exit 137 means the kernel killed it for memory. That is the first thing to check.
  • Steady climb is a leak. A single spike is a limit that is too small.
  • Set the limit from the measured peak plus headroom, not from a round number.

Try it: read a sample memory report and decide what killed the container.

# python3 oom-check.py
samples = [("10:00", 210), ("10:05", 320), ("10:10", 430), ("10:15", 510)]
limit_mb = 512
for when, used in samples:
    print(when, used, "MB of", limit_mb, "->", "OOM RISK" if used >= limit_mb * 0.9 else "ok")
print("pattern: steady climb, so this is a leak, not a spike")

Expected: the last two samples flag OOM risk, and the steady climb says leak. A leak gets fixed in the app. A bigger limit only buys time.

Master — A setup you can defend

Definition: At this level I can defend a production setup in a review: the data lives in named volumes, the secrets come from a store and never land in an image, every container has a memory and CPU limit, every service has a healthcheck that can go red, nothing runs as root, every image is distroless where it can be, and the scan gate blocks a critical flaw. I can prove each claim with one command. And I rehearse the worst day: a secret leaks, a container eats the host, a deploy is blocked, and the team knows what to do.

Types:

  • Named volumes, which hold the data that matters.
  • Secrets from a store, which never land in git or in an image.
  • Limits on every container, which keep one bad app from taking the host.
  • A scan gate that blocks, which proves it is a gate and not a report.

Uses: I use this to review a setup I did not build: I ask for the five proofs, and I watch them run. It is also how I teach the team: one bad setup, reviewed in public, teaches the controls better than a page of rules. Try it now: pick any production setup you own and run the five proofs. If one fails, that is the first gap to close.

Dimensions: At a hundred times the containers, the controls are not the bottleneck, the blast radius is. What breaks is the single host every container shares, or the single secret every service uses. It costs separate hosts, separate secrets, and a rehearsed leak, not more controls.

Rotate the secret without restarting the app. A secret that needs a restart to rotate is a secret that causes a deploy every time it changes. I mount secrets as files, and the app reads the file on each use, or watches it for changes. The rotation is: write the new secret to the store, the new file appears in the container, the app picks it up. No restart, no deploy, no downtime. I test it by rotating a test secret and watching the app use the new one while it keeps serving.

  • Secrets as files, read on use, never as environment variables the process keeps forever.
  • The app re-reads the file, so rotation needs no restart.
  • Test the rotation on a test secret first. A rotation that needs a restart is a rotation that causes a deploy.

Try it: check whether a running container would see a rotated secret.

# python3 secret-rotation.py
# Env vars are read once at start. Files can be re-read on each use.
import os
os.environ["DB_PASS"] = "old-secret"
open("/tmp/db-pass.txt", "w").write("old-secret")
print("env var at start:", os.environ["DB_PASS"])
open("/tmp/db-pass.txt", "w").write("new-secret")
print("env var after rotation:", os.environ["DB_PASS"], "(process never re-reads it)")
print("file secret after rotation:", open("/tmp/db-pass.txt").read(), "(app reads the file on each use)")

Expected: the env var stays old, the file secret updates. That is why secrets belong in files, not in the environment.

Go deeper

Start with the full production override below. It caps the web service, runs it as a non-root user, keeps it read only, mounts its secret as a file, and checks its health on a schedule. Do this: save it as compose.prod.yaml next to the compose file from Lesson 144, and run docker compose -f compose.yaml -f compose.prod.yaml config. Expected output: the config check prints the merged file, with the limits, the secret, and the healthcheck in place.

docker compose -f, flag by flag. Compose reads files. -f says which files, and the order says which one wins.

  • -f compose.yaml -f compose.prod.yaml names the compose files, in order. Repeat the flag to layer files. The later file wins where they disagree. Without the second -f, the prod overrides never load, and the stack runs with dev settings.

Flags you will meet next: docker compose --project-name myapp names the project, so two copies of the same stack can run side by side.

Try it: run docker compose -f compose.yaml -f compose.prod.yaml config and read the merged output. Then run it with only the first -f. The prod overrides disappear. The order of the flags is the config.

services:
  web:
    image: myapp-web:1.0.0
    user: "10001"
    read_only: true
    tmpfs:
      - /tmp
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "1.0"
    secrets:
      - db_password
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 3s
      retries: 3
secrets:
  db_password:
    external: true

curl -f, flag by flag. This line lives inside a HEALTHCHECK. Its job is to fail loudly when the app is broken, so the runtime can see it.

  • -f fails the command on a 4xx or 5xx status and prints no body. Without it curl exits 0 even on a 404, and a healthcheck passes on an error page. That is the exact failure -f exists to catch.

Flags you will meet next: --retry 3 retries a failed request, which is for a client, not a healthcheck.

Try it: run it against a healthy endpoint, then against a missing path. The healthy one exits 0. The missing one exits 22, and prints nothing. That exit code is what the healthcheck reads.

What the snippet means: the web runs as user 10001, its filesystem is read only except for a small tmpfs at /tmp, it may use at most 512 MB of memory and one CPU, its database password comes from an external secret, and its health is checked every 30 seconds. The secret is external, so it lives outside the file, and the file names it without holding its value.

Prove the secret is sealed. Do this: run docker compose -f compose.yaml -f compose.prod.yaml up -d, then run docker inspect web --format '{{json .Config.Env}}'. Expected output: no password in the list. Then run docker compose logs web. Expected output: no password in the logs either. If the password shows up in either place, it came from an env var, and the fix is the secret mount, plus a rotation, because the old password is already exposed.

docker compose logs, flag by flag. Logs are per service here, not per container. The service name is the filter.

  • docker compose logs api shows the logs of one service. Without the service name I get every service mixed together, and the error I want hides in the middle.

Flags you will meet next: -f follows the logs, and --tail 50 shows the last fifty lines.

Try it: run docker compose logs api, then run it with no service name. The first shows one service. The second shows every service mixed together, and the error hides in the middle.

docker inspect --format, flag by flag. inspect prints everything. --format prints one thing. I use the flag when I already know what I am looking for.

  • --format prints one value from the JSON instead of the whole document, using a Go template. Without it inspect prints pages of JSON, and the one field I want is somewhere in the middle.
  • {{json .Config.Env}} prints a whole block as JSON instead of one field. I use it when I want the full list of environment variables, not one of them.

Flags you will meet next: {{json .Config.Env}} prints a whole block as JSON, and {{.Id}} prints the image ID alone.

Try it: run docker inspect with no --format and pipe it to wc -l, then run it with --format '{{.State.Health.Status}}'. The first prints hundreds of lines. The second prints one word: healthy, unhealthy, or starting.

docker compose up -d, flag by flag. up builds the stack and starts it. -d is the flag that gives me my terminal back.

  • up -d creates and starts the services. -d detaches, so the prompt comes back at once. Without -d the logs fill the terminal, and the stack stops when I close it.

Flags you will meet next: --build rebuilds the images before starting, and --force-recreate recreates the containers even when nothing changed.

Try it: run up -d, then run docker compose ps. The services should show as running. Then run up without -d in a scratch terminal and watch the logs fill it. Stop it with Ctrl-C and note the stack stops with you.

Prove the limit holds under load. Do this: run docker stats --no-stream while the app is busy. Expected output: the MEM column stops at 512 MB, and the app slows instead of eating the host. Then run docker stats --no-stream on a container with no limit. Expected output: the MEM column keeps growing. That is the difference a limit makes, and it is visible in one command.

Prove the scan gate blocks. Do this: scan the image with your scanner of choice, read the list of known flaws, and update the base image to a version without the worst one. Expected output: the next scan is clean, and the deploy goes through. Then push an old image with a known critical flaw on purpose in a scratch registry, and watch the gate block it. A gate you have never seen block is a gate you have not tested. Practice all six controls in the Docker labs.

At 10x the containers, the shape holds and the numbers grow. At 100x, the design changes: the hosts split, the secrets split per service, and the scan gate runs in the pipeline, not on a laptop. The six controls never change. The discipline around them does. Lesson 48 is where containers become Kubernetes, and every control in this lesson shows up again there, at cluster scale.

Size the limits from measured use, not from round numbers

A memory limit of 512 MB is a guess. A limit set from the measured peak plus 30 percent is a decision. I run the container under real traffic, record the peak memory and the peak CPU, and I set the limits from those numbers. Then I watch for a week: if the container never gets near the limit, the limit is too high and the host is wasting space. If it hits the limit, the limit is too low and the app is being killed. The numbers decide, and I re-measure after every release.

# python3 limit-size.py
peak_memory_mb, peak_cpu = 380, 0.7
memory_limit = int(peak_memory_mb * 1.3)
cpu_limit = round(peak_cpu * 1.3, 2)
print("measured peak memory (MB):", peak_memory_mb, "-> limit:", memory_limit)
print("measured peak CPU:", peak_cpu, "-> limit:", cpu_limit)
print("run as: docker run --memory=%dm --cpus=%.2f my-app" % (memory_limit, cpu_limit))

docker run --memory and --cpus, flag by flag. Limits are a promise to the other containers on the host. Without them one container can take everything, and the rest starve.

  • --memory=512m caps the container memory at 512 megabytes. Without it the container can take all the host memory. Past the cap the kernel kills the process, and the log says OOMKilled.
  • --cpus=1.0 caps the container CPU share at one core. Without it the container can take every core on the host, and one greedy container starves the rest.
  • -d runs the container in the background and prints its ID, then returns the prompt. Without it the terminal stays attached to the app, and closing the terminal stops the container.
  • --name hello names the container. Without it Docker picks a random name, and every later command has to use the ID or that random name.

Flags you will meet next: --memory-swap caps memory plus swap together, and --cpuset-cpus pins the container to specific cores.

Try it: run a container with --memory=512m, then read the limit back with docker inspect --format '{{.HostConfig.Memory}}'. It should print 536870912, which is 512 megabytes in bytes.

Expected: memory limit 494 MB, CPU limit 0.91. Those numbers come from the measurement, and they change when the measurement changes.

One time it broke

What broke

A database password was in the logs. The app printed its environment on startup, for debugging, and the password was an environment variable, so it printed too. The logs went to a shared log store, where anyone on the team could read them. Nobody noticed for a week. The password worked, the app worked, and the leak was silent until a routine log search turned it up.

Timeline

  • 09:00,the app was deployed with the database password in an environment variable.
  • 09:01,the app printed its environment on startup, including the password, into the logs.
  • 09:05,the logs shipped to the shared log store, where the whole team could read them.
  • 16:20,a routine search for the word password in the logs turned up the value, in plain text.
  • 16:25,I rotated the password at once, and moved it to a secret mounted as a file.
  • 16:40,I removed the startup print of the environment, and the logs went clean.

What changed after

Three things. First, no secret ever travels as an environment variable. It comes from a secret store and is mounted as a file. Second, no app prints its environment on startup, ever. Third, I rotate a password the moment it appears anywhere it should not, because a leaked password is already used. Try it now: search the logs of any app you own for the word password. If you find a value, rotate it before the next deploy.

If you remember nothing else

Build it yourself

Run a working production setup for the four-service app from Lesson 144, with all six controls in place: named volumes, secrets from a store, limits on every container, healthchecks, non-root users, and a scanned image. It passes when all five proofs pass: the data survives, the secret is sealed, the limit holds, the healthcheck can go red, and the scan gate can block.

Build it, step by step

  • Write a compose.prod.yaml override with deploy limits of 512M and 1.0 CPU, user 10001, read_only true, a tmpfs at /tmp, a secrets block, and a healthcheck, for the web service.
  • Start the app with docker compose -f compose.yaml -f compose.prod.yaml up -d, and confirm all four services show in docker compose ps.
  • Prove the data survives: write a row into the database, run docker compose down and up again, and read the row back.
  • Prove the secret is sealed: run docker inspect on the web container and confirm no password shows in the env list, and search the logs for the password and confirm it is absent.
  • Prove the limit and the healthcheck: run docker stats under load and confirm the memory stops at 512 MB, then break the health endpoint in a scratch copy and confirm the status flips to unhealthy.

What success looks like

Success looks like this: all four services show in ps, with db healthy. The row survives the restart. Inspect shows no password. The logs show no password. Docker stats stops at 512 MB. The broken endpoint flips the status to unhealthy. If any proof fails, that control is missing, and the setup is not done.

If it fails, check this

If the data is gone, the volume is unnamed or the down passed -v. If the password shows in inspect or logs, it came from an env var: move it to a secret and rotate it. If the memory keeps growing past 512 MB, the limit is on the wrong service or the override file was not passed. If the status never flips, the healthcheck command is wrong or the endpoint does not exist.

Next in this series
Lesson 48 — Kubernetes Core