Platform Engineering · Lesson 47

Containers and Docker

I used to ship code by copying files onto a server and hoping the server had the right libraries. Sometimes it did. Sometimes it did not, and I found out at 2 AM. A container changed that deal for me. I pack the code, the runtime, and the libraries into one image. The image runs the same on my laptop and on the server. This lesson walks that image from my laptop to a running container, layer by layer, and shows where the time goes and where the size comes from.

Difficulty: Advanced · Know this first: Lesson 33

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in seconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most slow deploys I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.

Try it: shrink the image and watch the deploy get faster

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the build, the push, the pull, and the start change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the size of the image decides how fast the deploy goes.

150

The base is the bottom layer everything else stacks on. A slim base is about 5 to 50 MB. A full operating system base is 500 MB or more.

6

One layer per Dockerfile line that changes the filesystem. More layers mean more chances for the cache to miss, and each layer adds 20 MB in this model.

80

Share of layers the build reuses instead of rebuilding. A reused layer costs about 1 second. A rebuilt layer costs about 12 seconds.

Try this: (1) Set base to 1000, layers to 20, cache to 0. The image is 1480 MB, the build is about 250 seconds, the pull is about 123 seconds, and the total is about 475 seconds. (2) Set base to 5, layers to 3, cache to 100. The image is 145 MB, the build is about 13 seconds, the pull is about 12 seconds, and the total is about 53 seconds. Same app. Only the image changed.

The image as a stack of layers, from scratch

Extra diagram. I read the picture first: the boxes are the system, the moving dot is an image, and each step adds one piece until the full picture is there. No text needed to follow it.

One layer per Dockerfile instructionReused layer costs about 1 s, rebuilt about 12 sEach layer adds about 20 MB
123456789

BuildShipRunMoving dots: request · response

Rebuild with a warm cache: what rebuilds, what reuses

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a rebuild, and each step adds one piece until the full picture is there. No text needed to follow it.

Order the Dockerfile slowest and most stable firstPin the base by digestIgnore stray files with .dockerignore
1234567

ReusedRebuiltCache killerMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere, this is just you against the material.

Basic to master

The lesson above walks one image from my laptop to a running container, layer by layer. This section covers the same image four times, from the first build to the level where images run a real pipeline.

Basic — One image, same everywhere

Definition: I used to ship code by copying files onto a server and hoping the server had the right libraries. Sometimes it did. Sometimes it did not, and I found out at 2 AM. A container changed that deal. I pack the code, the runtime, and the libraries into one image. The image runs the same on my laptop and on the server.

Types:

  • The image — the packed box: code, runtime, libraries, in layers.
  • The container — the image, running. One image, many containers.
  • The Dockerfile — the recipe that builds the image, one instruction per layer.
  • The registry — where images live, so any machine can pull them.

Uses: I use this level every time I ship. Build the image once, run it anywhere. If it works on my laptop, the image is the same on the server, and the difference between the two is the environment, not the code.

Dimensions:

  • Load: At one service, one image is a few hundred megabytes and a fast pull. At fifty services, the registry holds thousands of images, and the pulls add up on every deploy.
  • Breaks: It breaks when the image is big and the pull is slow, or when the build leaves junk in the image. I notice it when deploys take minutes and the container is still pulling.
  • Cost: The cost is disk and time. Every layer is stored, every image is pulled, and a careless Dockerfile makes both bigger than they need to be.
  • Trade-off: I give up the speed of copying files to get the same thing everywhere. A copy is fast and it lies. An image is slower and it tells the truth.

Intermediate — Layers, caching, and size

Definition: An image is layers, and each Dockerfile instruction is one layer. The build cache reuses a layer when nothing above it changed. Order matters. Dependencies first, code last. The code changes every day. The dependencies change once a month.

Types:

  • Layer caching — stable, slow layers at the top, fast-changing code at the bottom.
  • The .dockerignore file — stray files out of the build, so a 2 GB test data folder does not invalidate the cache.
  • Multi-stage builds — build in one image, ship in another, so the tools never reach production.
  • Pinned base images — pin by digest, not by a moving tag, so next month's rebuild uses the same base.

Uses: I use this level when the build is slow. I order the Dockerfile, I add the ignore file, and I watch the build time fall. The fastest build is the one that reuses every layer except the code.

Dimensions:

  • Load: At one build a day, a slow build is an annoyance. At fifty builds a day, a slow build is a queue, and the queue is the deploy.
  • Breaks: It breaks when the build context is 2 GB because nobody ignored the test data folder, or when a moving tag pulls a new base and the build changes under me. I notice it when the build that worked yesterday fails today and nothing in the code changed.
  • Cost: The cost is build time and registry storage. Every layer is stored. A slim image is cheaper to store, faster to pull, and faster to push.
  • Trade-off: I give up the convenience of a moving tag to get a rebuild I can trust. Pinning by digest is more work. It is also the only way the same Dockerfile builds the same image twice.

Advanced — Security: scanning, distroless, and secrets

Definition: An image is code I did not write, running with my code. The base image has packages. The dependencies have packages. Some of them have known vulnerabilities. I scan the image in the pipeline, before it can be pulled into production, and I treat a critical finding the way I treat a failing test. The build stops.

Types:

  • Image scanning — the pipeline checks the base and the dependencies for known bad packages.
  • Distroless images — the runtime stays, the shell and the package manager go. An attacker who gets in finds no shell to use.
  • The debug image — the same app with a shell, for the rare times I need to look inside.
  • Secrets out of the image — no keys in the Dockerfile, no keys in the layers. A secret in a layer is a secret in the registry forever.

Uses: I use this level when the image reaches production. I scan on every build, and I scan the base on a schedule, because a clean image today can gain a known vulnerability tomorrow when a new CVE is published.

Dimensions:

  • Load: At one image, scanning is seconds. At hundreds of images, the scan results need a policy, or the team drowns in findings and stops reading them.
  • Breaks: It breaks when the scan is a warning nobody reads, or when a distroless image leaves me with no shell at 2 AM and no logs to read. I notice the first when a known vulnerability ships, and the second when I cannot debug the container that is failing.
  • Cost: The cost is pipeline time and the discipline to stop the build. A critical finding that does not stop the build is a report, not a control.
  • Trade-off: I give up the convenience of a shell in every container to get a smaller attack surface. Debugging gets harder. The logs have to be good, because the shell is gone.

Master — Images at scale in a real pipeline

Definition: At this level the image is a product with a lifecycle. Images are built by the pipeline, scanned, signed, stored, cleaned up, and pulled by machines that never see a human. The build host is the thing I watch, because every build leaves layers behind, and months of layers fill a disk at 2 AM.

Types:

  • The signed image — the build signs the image, and the cluster only runs signed images.
  • The cleanup job — old images and old layers removed on a schedule, so the build host never fills up.
  • The base image pipeline — one team owns the base, patches it, and every service rebuilds on it.
  • The size budget — the image has a size limit, and the pipeline fails the build that breaks it.

Uses: I use this level when the pipeline builds a hundred images a day. The cleanup job is the lesson I learned the hard way: builds started failing on the shared host, always with a disk full error, always late at night. The code was fine. The pipeline was fine. Nobody had ever cleaned up old images and old layers.

Dimensions:

  • Load: At this scale the registry is a service with its own SLO. A slow registry is a slow deploy for every team, and a full build host stops every build at once.
  • Breaks: It breaks when the cleanup never runs, or when the base image pipeline stalls and every service ships a base with a known vulnerability. I notice the first in the disk graph, and the second in the scan report.
  • Cost: The cost is the registry, the scanning, the signing, and the cleanup. It is a platform, and it needs an owner.
  • Trade-off: I give up letting every team build its own base to get one base that is patched, scanned, and trusted. Freedom is fifty bases. Safety is one.

Go deeper

Layer caching is the first trade-off. Order the Dockerfile so the slowest, most stable layers sit at the top. Dependencies first, code last. Copy only the dependency list, install, then copy the code. Pin the base image by digest, not by a moving tag, so a rebuild next month uses the same base. Use a .dockerignore file so stray files do not invalidate the cache or leak into the image. I have seen a build context of 2 GB because nobody ignored the test data folder.

Image scanning is the second trade-off. Scan the image in the pipeline, before it can be pulled into production. A scan finds known bad packages in the base and in the dependencies. I treat a critical finding the same way I treat a failing test: the build stops. I also scan the base image on a schedule, because a clean image today can gain a known vulnerability tomorrow when a new CVE is published.

Distroless images are the third trade-off. A distroless image keeps the runtime and drops the shell, the package manager, and the extra tools. The app still runs. An attacker who gets in finds no shell to use and no package manager to install with. The cost is debugging: I cannot shell into the container to look around, so I need good logs and a debug image for the rare times I must.

Rootless containers are the fourth trade-off. By default a process in a container can run as root inside the container, and that root is safer than host root but not harmless. Rootless mode runs the container as a normal user, so a breakout lands on a user with no privileges instead of on root. The cost is setup. Some tools expect root, and port binding below 1024 needs a workaround. I accept that cost for anything facing the internet.

At 10x the scale, one registry is not enough, so I mirror it close to the hosts. Image size matters more, because ten hosts pulling at once is ten times the traffic. I set an image size budget and fail the build when it grows past it.

At 100x, the image becomes a contract. Every team builds from the same approved base images, the same scan gate, the same tags. A team cannot ship a base image nobody reviewed. The runtime enforces signed images only, so a host refuses to run an image that did not come through the pipeline.

# Stage 1: build. The tools live here and stay here.
FROM node:20-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build

# Stage 2: run. Only the built app and its runtime come along.
FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER nonroot
EXPOSE 3000
CMD ["dist/server.js"]

What the snippet means: the first stage has the compiler and the build tools. The second stage copies only the built output and the runtime dependencies, and it runs as a non-root user with no shell. The build tools never reach production, so the image is smaller and there is less for an attacker to use.

One time it broke

What broke

Builds started failing on the shared build host, always with a disk full error, always late at night. The app code was fine. The pipeline was fine. The host had run out of disk because nobody had ever cleaned up old images and old layers. Every build left layers behind. Months of them.

Timeline

  • 22:10the first build failed with no space left on device. I thought it was a one-off and reran it.
  • 22:14the rerun failed the same way. The error was on the build host, not in the app.
  • 22:20I checked the disk. 98 percent full. The image store was most of it, full of images from builds nobody needed any more.
  • 22:35I removed dangling images and layers older than 30 days. The disk dropped to 55 percent full.
  • 22:50builds passed again. The deploy went out an hour late.
  • 09:00next morning, I checked the other build hosts. Two of them were above 85 percent full too.

What changed after

Three things. First, a nightly prune job that removes unused images older than 14 days, with a dry run report before it deletes anything. Second, a disk alert at 70 and 85 percent on every build host, so the next fill-up pages someone during the day. Third, an image size budget in the pipeline: an image over 500 MB fails the build and makes someone explain why. Old layers are not free storage. They are a slow leak, and the leak only shows up when a deploy is already late.

If you remember nothing else

Design exercise

Design the container image for a small API that serves 500 requests per second. It must start in under 5 seconds, the image must stay under 300 MB, and it must run as a non-root user. The app is written in Node, it needs three environment variables to start, and it reads a config file at boot. You have a registry, a build host, and a fleet of hosts that pull the image.

Clarifying questions to ask

  • How big are the dependencies, and are dev dependencies included by mistake?
  • What is in the base image today, and does the app use any of it?
  • How often does the image change, and how many hosts pull it at once?
  • What happens when a required environment variable is missing: does the app fail fast with a clear error, or start broken?
  • Where does the config file live, and is it baked into the image or mounted at run time?

Rough sizing

Start with the simulator numbers. A slim base of about 50 MB, 6 layers at 20 MB each, plus 80 MB of app and dependencies gives about 250 MB, which fits under the 300 MB budget. Build time at 80 percent cache is about 29 seconds. Pull time at 12 MB per second is about 21 seconds. Start time is about 3 seconds, which fits under the 5 second limit. If the base were 500 MB instead, the image would be about 700 MB, the pull would take about 58 seconds, and the start would take about 6 seconds. Same app, failed budget.

Trade-offs to name

A distroless base buys a smaller image and no shell, and it costs easy debugging. Baking the config into the image buys a self-contained artifact, and it costs a rebuild for every config change. Mounting the config at run time buys fast config changes, and it costs a second thing that can be wrong at start. Failing fast on a missing environment variable buys a loud error at deploy time, and it costs a container that will not start until the config is right. That is the right cost to pay.

Next in this series
Lesson 48 — Kubernetes Core