System Design · Lesson 104

Helm

Lesson 48 taught you what Kubernetes runs. This lesson teaches you how a team ships it without hand-writing fifty YAML files per service. Helm is a package manager for Kubernetes. A chart is a bundle of templates plus default values. A release is one installed copy of that chart, with a name, a revision number, and a record of what it installed. One command deploys the checkout app, and Helm does the rest: it fills in the values, renders the templates into plain Kubernetes manifests, checks them against what is already running, applies them in order, runs the hooks, and writes down the release so it can be rolled back. I learned Helm the way I learn most tools in this job. I shipped a values file with the wrong image tag, watched a good deploy roll out a broken build, and then read the release history until the release confessed. This lesson walks one deploy through Helm, step by step, from a first chart to the deep end: chart design, Helm against Kustomize, OCI registries, and rollback.

Difficulty: Advanced · Know this first: Lesson 48, Lesson 50

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one Helm deploy, not a measurement from a live system, but the shape is honest: in most deploys I have chased, Helm's own work is small and the rollout is the bill. Step through the lesson or switch the scenario and watch the bar change. The bad values deploy triples the trip without Helm doing anything different in kind. Same chart, same templates. Only the image tag and the hooks changed, and the release failed anyway.

Try it: size the chart and watch the deploy bend

The budget bar above shows three fixed stories. This simulator lets you write your own. It models one Helm deploy, from rendering the chart to the pods coming up. Move a slider and watch the render time, the apply time, and the rollout change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: templates and values are cheap, the rollout is the bill, and every replica you add is paid for twice, once at apply and once at rollout.

4

Each override adds 1.5 ms to rendering. At the default of 4, that is 6 ms of the render time.

25

Each template costs 0.4 ms to render. At the default of 25, that is 10 ms of the render time.

3

Each replica adds 6 ms at apply and 14 ms at rollout. At the default of 3, apply is 30 ms and rollout is 42 ms.

1

Each hook adds 8 ms. At the default of 1, hook time is 8 ms.

Try this: (1) Leave the defaults: 4 overrides, 25 templates, 3 replicas, 1 hook. The total is about 111 ms. (2) Push the replicas to 20 and the templates to 100 with 20 overrides. The total is about 537 ms. Same chart, same deploy. Only the knobs changed. (3) Set everything back, then set the replicas to 1 and watch the rollout shrink to 14 ms. That is what a small service costs to deploy, and it is why small services are cheap to ship.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Helm looks like one command. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: install a chart and see a release, then write your own values, then design a chart a team can reuse, then run charts as the unit of delivery across environments. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.

Basic — Install a chart, read a release

Definition: Helm at the basic level is a package manager for Kubernetes. A chart is a bundle of templates and default values. A release is one installed copy of that chart, with a name and a revision number. You run helm install, Helm renders the templates with the values, applies the manifests, and records the release.

Types:

  • A chart from a public repository, installed as-is.
  • A release name, the one name you give the installed copy.
  • The release history, one revision per install or upgrade.
  • helm status and helm list, the two commands that tell you what is installed and whether it is healthy.

Uses: when a team needs a database or a tool running in a cluster and does not want to write the YAML by hand. One command installs a chart, and the release history means the install can be repeated or removed cleanly.

Dimensions: Scale: holds up to a handful of charts installed by hand. Failure modes: a release installed with the wrong name is a different release, and Helm will not find it under the name you meant; a chart installed from a repo that moved is a chart that cannot be found. Cost: Helm 3 has no server component, so there is no extra process to run; the cost is the release history, one Secret per revision in the cluster. Trade-offs: installing a chart as-is is fast and opaque. You get a running app and no understanding of what it installed, so the first time it breaks you are reading someone else's templates.

Intermediate — Your own values, your own chart

Definition: the intermediate layer is values and chart authoring. Values are the config that fills the chart's templates, merged in layers: chart defaults, then environment values files, then --set flags. A chart you write yourself has a Chart.yaml, a values.yaml with sane defaults, templates for each object, and a values schema that says what types the values must be.

Types:

  • values.yaml, the chart defaults.
  • Environment values files, one per environment, overriding the defaults.
  • --set flags, the last layer, for one-off overrides.
  • Named templates and helpers, shared snippets that keep the chart from repeating itself.
  • A values schema, a JSON Schema file that catches type errors at render time.
  • helm template, the command that renders the chart on your laptop without touching a cluster.

Uses: when a team ships its own service and wants staging and production to differ by values, not by a second copy of the YAML. One chart, two values files, and the only difference between the environments is written down in a file I can read.

Dimensions: Scale: holds to a few dozen charts in one repo, but the values files grow fast, because every environment adds a layer and every override adds a line. Failure modes: a --set flag that overrides a values file silently; a values schema that is missing lets a string where a number should be sail through to the cluster; a template that renders an empty image tag deploys a pod that pulls nothing. Cost: maintenance; each chart needs an owner, each values file needs a review, and the templates need tests. Trade-offs: a chart you write is a chart you own. It fits your service exactly and it costs you the upkeep. A public chart is free and fits nobody exactly, so you pay in values overrides instead.

Advanced — Hooks, rollback, OCI, and Helm against Kustomize

Definition: the advanced layer is the deploy lifecycle. Hooks run Jobs at fixed points in the deploy: migrations before an upgrade, smoke tests after one. Rollback restores a previous release revision, re-rendering its values and re-applying its manifests. OCI registries store charts as container images, so a chart moves through the same registry and the same signing as the app image. Kustomize is the other way to manage Kubernetes YAML: it patches base manifests with overlays instead of templating them, and the choice between Helm and Kustomize is a choice about where the variation lives.

Types:

  • Pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, and pre-rollback hooks.
  • helm rollback, which restores a previous revision.
  • OCI registries for charts, with helm push and helm pull.
  • Chart signing and provenance, so a chart can be verified before it is installed.
  • Kustomize overlays, the patch-based alternative.
  • ArgoCD and Flux running Helm charts from Git, the GitOps shape from Lesson 50.

Uses: when a deploy has to be safe, not just possible. A migration hook that must run before the new code starts. A rollback that must work at 2 AM without anyone re-reading the chart. A chart stored in the same registry as the image, signed, so the cluster only installs what the pipeline built.

Dimensions: Scale: the chart count becomes the bottleneck; every chart needs a version, a registry entry, and an owner, and the release history grows with every deploy. Failure modes: a hook that fails blocks the release, and a hook that deletes data destroys the thing it was meant to update; a rollback that restores the old manifests but not the old data leaves the users staring at the bad data; a chart in an OCI registry that was never signed is a chart nobody verified. Cost: operational; hooks need owners, registries need signing, and rollbacks need drills, because a rollback nobody has practiced is a rollback nobody trusts at 2 AM. Trade-offs: hooks buy gates and cost deploy time. OCI buys one registry for charts and images and costs a registry that has to be secured. Rollback buys a way back and costs the discipline of keeping the release history clean.

Master — Chart design for a platform: libraries, tests, and delivery

Definition: the master layer is chart design as platform work. A library chart holds shared templates that other charts include, so every service chart gets the same Deployment shape, the same labels, and the same probes. Chart tests run helm template and helm lint in the pipeline, and a rendered manifest is diffed before it is approved. Delivery means the chart is versioned, published, and consumed by GitOps, so a deploy is a pull request that changes a chart version, and the cluster converges on it.

Types:

  • Library charts, shared templates included by service charts.
  • Chart testing in CI: helm lint, helm template, and diff checks.
  • Versioning and publishing: semantic versions, changelogs, and a registry.
  • GitOps consumption: ArgoCD or Flux rendering the chart from Git.
  • Secrets handling: sealed secrets or an external secrets operator, because a chart must never hold a plain secret.
  • Chart deprecation and migration, the plan for retiring a chart version without breaking every service that uses it.

Uses: when a platform team serves dozens of service teams and every team needs the same shape of deploy. One library chart, one set of defaults, and a service team writes twenty lines of values instead of two hundred lines of YAML.

Dimensions: Scale: holds at hundreds of charts, because the cost moves from per-chart YAML to one library and a pipeline. Failure modes: a library chart change that breaks every service chart that includes it; a chart version published without a changelog leaves every team guessing what changed; a secret that slips into a values file is a secret in Git, and Git never forgets. Cost: expertise and process; library charts need an owner, the pipeline needs tests, and the versioning needs discipline. Trade-offs: a library chart buys consistency and costs flexibility, because every service gets the same shape whether it wants it or not. GitOps delivery buys an audit trail and costs a pull request for every deploy. Testing buys safety and costs pipeline time, and the time is worth it, because a chart change is a change to every service at once.

Go deeper

A good chart is small and boring. It has one job: install one service the same way every time. I keep the templates to the objects the service needs and nothing else. I put the variation in the values, not in if statements, because a template with ten conditionals is a template nobody can read. I give every object the same labels, the same annotations, and the same probes, because the day I need to find every pod from one chart, the labels are how I find them. And I write the defaults for production, not for a laptop, because the default is what a tired engineer at 2 AM will deploy, and the default should be safe.

Helm and Kustomize solve the same problem from opposite ends. Helm templates the YAML. You write a template with holes, fill the holes with values, and get a finished manifest. Kustomize patches the YAML. You write a base manifest, then an overlay that changes a few fields for each environment. I pick Helm when the variation is large: many environments, many values, a chart that ships as a package. I pick Kustomize when the variation is small: a base that is almost right, and an overlay that changes an image tag and a replica count. The honest answer is that most teams end up using both. Helm packages the app, Kustomize patches the last mile. The wrong move is picking one and forbidding the other, because the team then bends the tool to do a job it was never built for.

Charts used to live in chart repositories, which were a separate kind of server to run. Now they live in OCI registries, the same registries that hold container images. A chart is pushed, pulled, and versioned like an image, and the same registry that stores the app image stores the chart that deploys it. The win is one registry to secure instead of two. The extra step is signing. A signed chart can be verified before it is installed, and an unsigned chart is a chart nobody verified. I sign charts in the pipeline and verify them in the cluster, because a chart that installs a Deployment can install anything, and anything includes a bad image.

Rollback is the most misunderstood command in Helm. It restores a previous release revision: the old chart version, the old values, the old manifests. It does not restore the old data. If the bad deploy ran a migration, the migration stays. If the bad version wrote bad rows, the rows stay. And it does not restore a secret that was rotated. I drill rollbacks the same way I drill restores, because a rollback nobody has practiced is a rollback nobody trusts. And I keep the migration story separate from the chart story. Migrations go forward, never back, and the app code is written to read both the old and the new shape until the old shape is gone.

At 10x the charts, the problem is not Helm. It is the values. Every environment adds a values file, every team adds overrides, and the merged values for one deploy become a document nobody has read in full. I move the values into Git, review them like code, and render them in the pipeline so the diff is visible before the deploy. At 100x, the design changes. The chart stops being per-service and becomes a library, and the service teams write twenty lines of values instead of two hundred lines of YAML. The release history gets a retention policy, because a thousand revisions per release is a thousand Secrets in the cluster. And the deploy moves behind GitOps, so a deploy is a pull request, and the pull request is the audit trail.

# Chart.yaml — the chart, versioned separately from the app
apiVersion: v2
name: checkout
version: 3.2.0
appVersion: "2.4.1"

# values.yaml — the defaults, written for production
image:
  repository: registry.example.com/checkout
  tag: "2.4.1"
replicaCount: 3
resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

# templates/deployment.yaml — one hole per value, no conditionals
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

# values.schema.json — the guard, checked at render time
{
  "type": "object",
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 1 },
    "image": {
      "type": "object",
      "properties": {
        "tag": { "type": "string", "minLength": 1 }
      },
      "required": ["tag"]
    }
  },
  "required": ["replicaCount", "image"]
}

Four files, one deploy. The Chart.yaml names the chart and versions it separately from the app. The values.yaml holds the defaults, and the defaults are for production, because the default is what a tired engineer deploys. The Deployment template has one hole per value and no conditionals, so the rendered manifest is easy to read. The values schema is the guard: a replica count that is a string fails at render time, before the cluster ever sees it. Read them together and you can see the whole deploy: name it, value it, render it, guard it. Change one of them without reading the other three and you have changed the deploy in a way you did not intend.

One time it broke

This one stayed with me because every check Helm has passed, and the deploy was still wrong. The chart was fine. The values lied, and nothing in the pipeline read the diff. This is an illustrative scenario built from a real SRE pattern, anonymized, with no company names and no real system names.

What broke

A team deployed a checkout service with Helm. A values override in the deploy pipeline set the image tag to latest, because someone had added the override months earlier for a test and nobody removed it. The latest build that week had a bug that crashed the app on start. The chart rendered fine, the schema passed, the manifests were valid, and the diff showed the tag change plainly, but the deploy was auto-approved and nobody read the diff. The pods rolled out, crashed, and the Deployment kept retrying. The pre-upgrade hook ran a migration first, so the database was already on the new shape before anyone noticed the app was down. The release was marked failed, and the rollback restored the old pods, but the migration stayed, and the old code had to read the new database shape until a fix shipped.

Timeline

  • 09:40Deploy pipeline starts; override sets the image tag to latest; chart renders cleanly.
  • 09:41Diff shows image 2.4.0 to latest; deploy auto-approved; nobody reads the diff.
  • 09:42Pre-upgrade hook runs the migration; database moves to the new shape.
  • 09:44New pods start and crash; CrashLoopBackOff; Deployment keeps retrying.
  • 09:58First user report, checkout is down; on-call checks the pods, then the release history.
  • 10:05On-call finds the override in the pipeline and the tag in the merged values; the chart is innocent.
  • 10:12helm rollback to the last good revision; old pods come back healthy.
  • 10:20Old code reads the new database shape; it works, but only because the migration was written to be backward compatible.
  • 10:45Override removed from the pipeline; diff review added as a required step for production deploys.

What changed after

Three things. First, the image tag latest is banned in production values, and the pipeline rejects it. A tag must be a version number, and the version number must exist in the registry. Second, the diff is a required review step. A production deploy does not auto-approve anymore. A human reads helm diff before the apply, and the diff is attached to the deploy record. Third, migrations are written to be backward compatible, always, because a rollback restores the old code and the old code has to read the new shape. The chart did nothing wrong in this incident. The lesson is that a valid deploy can still be a wrong deploy, and the only guard that would have caught it was a human reading the diff.

If you remember nothing else

Design exercise

Design the Helm setup for a platform with 40 services across 3 environments: dev, staging, and production. Each service ships as a Helm chart. Deploys must be reviewable before they apply, roll back at 2 AM without anyone re-reading the chart, and never ship the image tag latest. One service needs a database migration on every deploy. The latency budget is not the question here. The question is the deploy: sketch the chart layout, the values layers, the hooks, the registry, and the rollback story, and say what you would measure.

Clarifying questions to ask

  • Which values differ per environment, and which are the same everywhere: is the difference a file, or a whole second chart?
  • Who owns the library chart, and what happens when it changes: does every service deploy again, or does the change wait for the next deploy?
  • What happens when a migration fails halfway: does the deploy stop, does the release roll back, and who owns the half-migrated database?
  • How is a chart versioned and published: does a chart change ship with the app, or on its own schedule?
  • What does the rollback drill look like: has anyone actually run helm rollback in staging this month, and how long did it take?

Rough sizing

Start with the budget from this lesson. One deploy takes about 80 ms of Helm-side work: render 5, validate 4, diff 6, apply 12, hooks 8, rollout 40, record 2, plus the values merge. Forty services deploying once a day is 40 deploys a day, and the Helm-side work is trivial next to the rollouts. Little's Law for the sizing: at 5 deploys an hour in the busy window, each spending 0.08 seconds in Helm-side work, the number inside the system at once is 5 times 0.08, which is less than 1. Helm is never the bottleneck. The bottleneck is the rollout, and the rollout is the Deployment controller's job. I size the pipeline for the rollout, not for Helm: at 3 replicas per service and 14 ms per pod to become ready, one deploy costs about 42 ms of rollout, and 40 services in parallel cost the same 42 ms if the cluster has room. The number I watch is the release history: 40 services times 30 deploys a month is 1,200 revisions a month, and each one is a Secret in the cluster, so I set a retention policy and clean the history.

Trade-offs to name

Helm buys a package and costs a template language to learn. Kustomize buys a patch and costs a base that has to stay almost right. A library chart buys consistency and costs flexibility, because every service gets the same shape. Hooks buy gates and cost deploy time, and a hook that deletes data is a deploy that can destroy the thing it was meant to update. OCI buys one registry and costs a registry that has to be secured and signed. And the target is reviewability, not speed. A deploy that is fast and unreadable fails this design, and the diff is where the review lives.

Next in this series
Lesson 105 — Kubernetes Networking & CNI

This lesson packaged the app and shipped it. The next one follows the request after the deploy: how pods find each other, how a Service sends traffic to the right pod, and what the CNI plugin does underneath it all.

Go to Lesson 105 →