Lesson 109 gave you Terraform. It builds things from files, and it remembers what it built in a state file. This lesson is about that memory. Where it lives, who may touch it, what happens when two people apply at once, and what to do when the real world drifts away from the files.
Difficulty: Advanced · Know this first: Lesson 109 — Terraform
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with one file. Add one box at a time until the whole loop is on the map, lock included.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the plan, the apply, and the state calls change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: drift and a slow backend bend the whole apply.
How many resources the files describe. The plan costs about half a millisecond per resource, and the apply costs about one millisecond per resource. A bigger stack is a slower plan before anything goes wrong.
Resources someone changed outside the files. Each drifted resource adds about 12 ms to the plan and about 35 ms to the apply, because the apply has to rebuild it. Two drifted resources cost about 94 ms. Twenty cost about 940 ms.
How slow each call to the state backend is. Terraform makes about four state calls per run, so the cost is four times this number divided by 25. At 100 ms a call the state costs about 16 ms. At 500 ms it costs about 80 ms.
Try this: (1) Set resources to 100, drifted to 5, latency to 200. The plan is about 125 ms, the apply about 275 ms, and the total is about 472 ms. (2) Set all three to the worst end: 200 resources, 20 drifted, 500 ms latency. The plan is 355 ms, the apply is 900 ms, and the total is about 1,375 ms. Same files, same state. Only the inputs changed.
Type your own numbers: Get the total under 300 ms. Type 100 resources, 0 drifted, 20 ms latency, and run it. Then raise the drift until it fails.
Three questions. No score is saved anywhere — this is just you against the material.
I take the same state file four times here. First I learn the loop: files, state, cloud, plan, apply, state. Then I learn to keep the state where the team can reach it. Then I learn drift, imports, and surgery. Last I run it for the world.
Definition: Terraform keeps a state file that records what it built. The loop is simple: read the files, read the state, read the cloud, plan the diff, apply it, write the state back. The lock makes sure only one apply writes the state at a time. Local state lives on your disk. Remote state lives in a backend the whole team can reach.
Types:
Uses: I use remote state the moment a second person touches the same infrastructure. I use workspaces when dev, staging, and prod share the same files and differ only in a few variables. I check the lock before I force anything.
Dimensions: At low traffic everything is nearly free. What breaks first is the state file: deleted, corrupted, or written by two applies at once. It costs almost nothing to put the state in a backend with versioning and locking. It costs a rebuild when you skip that step.
Make a folder called tf-state. Save this file as main.tf. It creates one resource and keeps the state in a local file, so everything runs on your machine.
terraform {
backend "local" {}
}
variable "name" {
type = string
default = "web"
}
resource "null_resource" "box" {
triggers = {
name = var.name
}
}terraform init -backend=false
terraform validate
terraform fmt -check What you should see: Success! The configuration is valid. The fmt check prints nothing, which means the file is already formatted.
Read the flags: terraform init -backend=false
This init sets up the working directory without touching the backend. It is the safe way to learn, because no remote state is read or written.
-backend=false: This tells init to skip the backend entirely. Without it, a plain init would try to reach the remote backend in the config. While I am learning, that reach can fail or point at real state.-migrate-state (added): The lesson did not show this one yet. It moves state to a new backend when the backend block changes. Init asks first, and -force-copy answers yes in a script.Try it: open main.tf, delete the three-line terraform block that says backend local, save it as backend.tf, and run terraform init -backend=false with terraform validate. Expect the success line with no AWS credentials.
Read the flags: terraform validate and terraform fmt -check
Validate checks the config makes sense. Fmt checks it looks right. Both run without calling the cloud, so they belong in every pipeline before plan.
terraform validate: This checks syntax and references. It never calls the cloud and never reads live state. It needs init first, because it needs the providers installed.terraform fmt: This rewrites files into the canonical format. It changes files on disk, so I run it before I commit.-check: This flips fmt from rewriting to reporting. It prints the names of files that need formatting and exits non-zero. Without it, fmt would silently rewrite the files. In CI I want the report, not the rewrite.-diff (added): The lesson did not show this one yet. It prints the diff fmt would make, next to the file names. I use it with -check when I want to see the change before I accept it.-recursive (added): The lesson did not show this one yet. It formats subdirectories too. Without it, fmt only touches the current directory, and nested modules stay unformatted.Try it: run terraform fmt -check on a clean file, then add a badly spaced line and run it again. Expect silence first, then the file name and a non-zero exit.
It passes when: validate prints the success line and fmt prints nothing. The state file appears after the first apply in the Advanced section.
Try it: Basic
What does the state file record?
Now do it: Run the drill above: build the tf-state folder, run terraform validate, and expect the success line.
Definition: A backend is more than a bucket. It stores the state, versions every write, encrypts it at rest, and locks it during applies. The common pattern is an object store for the state and a lock table beside it. Terraform Cloud and similar services do the same job with a run queue on top.
Types:
Uses: I use one state file per environment, never one state file for everything. I use versioning so a bad write can be rolled back to the previous state. I use the pipeline to apply, so no state is ever written from a laptop.
Dimensions: As the team grows, the state file grows, and the apply slows down. What breaks is the shared state file: one team waiting on another team's lock, or one bad apply blocking everyone. It costs a split of the state by service or by team. The trade is that more state files mean more backends to look after.
This is the remote backend from the lesson. Open main.tf, delete the three-line terraform block that says backend "local", and save this file as backend.tf next to it. Validate checks the shape without ever touching AWS.
terraform {
backend "s3" {
bucket = "tf-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "tf-locks"
}
}terraform init -backend=false
terraform validate What you should see: Success! The configuration is valid.
It passes when: validate prints the success line. No AWS credentials are needed, because validate never calls AWS. Keep the -backend=false flag while you are learning: a plain terraform init would try to reach the bucket.
Try it: Intermediate
I split dev and prod. How many state files do I use?
Now do it: Run the drill above: move the backend block into backend.tf, run terraform validate, and expect the success line with no cloud credentials.
Definition: Drift is the cloud disagreeing with the files. Terraform detects it at plan time, and the apply removes it. Import brings a resource the console created into the state, so the files can own it. State surgery, mv and rm, moves resources between states without touching the cloud. Force-unlock releases a stuck lock, and it is the most dangerous command in this lesson.
Types:
Uses: I use import when a resource was created by hand and now needs to live in the files. I use state mv after a refactor, so the next plan does not destroy and recreate a live resource. I use force-unlock only after I prove, from the backend and from the team, that nobody is applying.
Dimensions: At ten times the resources, drift is constant, not rare. What breaks is the surgery: a wrong mv, and the next plan destroys a database it thinks is new. It costs a state backup before every operation and a plan after every operation. The trade is that surgery is powerful and unforgiving in the same breath.
Put the local backend back in main.tf and apply the config. Back the state up, then change one value and watch the plan notice. Then rename the resource in the state and watch the list change.
terraform apply -auto-approve
cp terraform.tfstate terraform.tfstate.20261120.bak
terraform state list
terraform state mv null_resource.box null_resource.web
terraform state list
terraform state mv null_resource.web null_resource.box
terraform state list Now the drift check. Change the trigger value, run plan, change it back, run plan again.
Read the flags: terraform apply -auto-approve
Apply is the moment Terraform commits. The flag removes the prompt, so the command is for pipelines and scratch work, never for a first run against prod by hand.
-auto-approve: This skips the yes or no prompt. Without it, apply shows the plan and waits for me to type yes. I never use it by hand on prod, because the prompt is the last chance to catch a destroy.-input=false (added): The lesson did not show this one yet. It stops Terraform from asking for a missing variable. A missing variable then fails the run with an error instead of waiting for input that never comes in a pipeline.Try it: run terraform apply -auto-approve in a scratch stack, then run it again with no changes. Expect the second run to report nothing to do.
Read the flags: terraform state list, terraform state mv, and cp
Before I move anything in state, I back the state file up. A copy is cheap. A lost state file is a rebuild of the whole stack by hand.
cp terraform.tfstate terraform.tfstate.20261120.bak: Cp copies the file. The date in the backup name tells me which copy is newest. I make the copy before any state surgery, not after.terraform state list: This lists every address in the state. Nothing is modified. I run it first, so the addresses I type into mv are copied, not guessed.terraform state mv: This moves one item in the state to a new address. The cloud object stays put. Only the record moves. A move that names a wrong address fails loudly, which is what I want.terraform state show (added): The lesson did not show this one yet. It prints one resource in full. I run it after a mv to prove the right object moved.Try it: run terraform state list, copy one address exactly, run state mv to a new name, and run state list again. Expect the old name gone and the new name present, with no plan changes.
sed -i 's/default = "web"/default = "web2"/' main.tf
terraform plan -detailed-exitcode; echo $?
sed -i 's/default = "web2"/default = "web"/' main.tf
terraform plan -detailed-exitcode; echo $? What you should see: terraform state list prints null_resource.box, then null_resource.web, then null_resource.box again. The drifted plan exits 2. The restored plan exits 0.
Read the flags: terraform plan -detailed-exitcode
A plain plan always exits 0 when it runs, even when it found changes. That makes it useless in a script. This flag turns the result into a number a script can read.
-detailed-exitcode: This changes the exit code. It is 0 when nothing changes, 1 on error, and 2 when changes exist. Without it, a script cannot tell a clean plan from a plan full of changes.-out=tfplan: This saves the plan to a file, so the same plan can be applied later. In a gate, I save the plan when the exit code is 2, and a human reviews it before apply.echo $?: The dollar question mark prints the exit code of the last command. I run it right after the plan, because any command in between overwrites it.-refresh-only (added): The lesson did not show this one yet. It refreshes state from the live cloud without proposing changes. Combined with -detailed-exitcode, it becomes a drift check that a scheduler can run every night.Try it: run terraform plan -detailed-exitcode, then echo $?. Expect 2 when changes exist and 0 after you apply them.
It passes when: the list shows each rename, the drifted plan exits 2, and the restored plan exits 0. Exit code 2 means a diff exists. Exit code 0 means no diff. The backup file is the undo button: make one before every state command.
The workspace that pointed at prod. Workspaces swap the state file, not the code. I have watched a plan run against prod because the terminal was still sitting in the prod workspace from yesterday. The plan output names the workspace, and I read that line first.
Do this: run terraform workspace show, then terraform workspace select staging, then run it again. Expected: the output names the workspace you are standing in, every time. Add the guard in the snippet so a plan against prod fails unless a variable says it is allowed.
Read the flags: terraform workspace show and terraform workspace select
A workspace is one state file per environment in the same config. Show tells me where I am. Select moves me. The wrong workspace applies dev config to prod state.
terraform workspace show: This prints the current workspace name. I run it before any apply, because the prompt does not always show it.terraform workspace select staging: This switches the state file the commands use. The config does not change. Only the state changes. A select to the wrong name is how prod gets dev values.terraform workspace list (added): The lesson did not show this one yet. It lists every workspace. I run it when I cannot remember the exact name, because select needs the name spelled right.terraform workspace new (added): The lesson did not show this one yet. It creates a workspace and switches to it. The new workspace starts with empty state, so the next plan shows everything as new.Try it: run terraform workspace show, select staging, run show again, and run terraform plan. Expect the plan to match staging, not dev.
variable "confirm_prod" {
type = bool
description = "Set to true to allow an apply against the prod workspace."
default = false
}
resource "terraform_data" "prod_guard" {
lifecycle {
precondition {
condition = var.environment != "prod" || var.confirm_prod
error_message = "Prod applies need confirm_prod=true."
}
}
}
What the snippet means: the precondition runs before anything applies. A prod apply without the flag fails with a plain message, and the flag is a deliberate choice, not just the right workspace.
Moving a resource into a module. A refactor that moves a resource into a module changes its address. The plan reads the new address as a new resource and the old address as a deletion. The fix is a state move, done before the next plan.
Do this: move a resource block into a module, run terraform plan, and expect a destroy and a create. Run the move command from the snippet, then run the plan again. Expected: the plan is empty, because the state now uses the new address for the same real object.
terraform state mv 'aws_instance.app' 'module.app.aws_instance.app'
What the snippet means: the state address follows the code address. Move the state first and the refactor is free. Move the code first and the plan wants to rebuild a live resource.
Try it: Advanced
I renamed a resource in the files. What keeps the next plan from destroying and recreating the live resource?
Now do it: Run the drill above: back the state up, run state list, run state mv, and expect the old name gone and the new name present.
Definition: At this level the state is a system of its own. Many state files, one per service, composed through remote state data sources. A module registry so teams share code instead of copying it. Policy checks on the plan before it applies. Drift detection that runs on a schedule and pages a human when the cloud moves.
Types:
Uses: I use this when more than one team applies to the same cloud account, or when a bad apply can take down a service other teams depend on. It is also how I review a setup I did not build: I ask where the state lives, who can write it, and what happens when it is gone.
Dimensions: At a hundred times the resources the state file is not the bottleneck, the blast radius is. What breaks is the single state file that everything depends on, or the lock that one stuck apply holds for an hour. It costs real operational work: state splits, lock monitoring, and restore drills. The trade is that the state is now infrastructure, and it needs an owner the way the database has one.
Make three folders, dev, staging, and prod. Copy main.tf into each one. Each folder gets this backend file with its own state path. Then prove each folder is valid on its own.
terraform {
backend "local" {
path = "../state/dev.tfstate"
}
} Change dev to staging and prod in the other two folders, then run this for each folder.
terraform init -backend=false
terraform validate
terraform fmt -check What you should see: Success! The configuration is valid. three times, once per folder, and fmt prints nothing.
It passes when: all three folders validate. Each folder has its own state path, so a dev apply can never touch the prod state file. That is the whole point of the split.
The restore drill for the state itself. A versioned backend can roll a bad state back. I prove that on purpose, in a scratch workspace, before I ever need it for real. A backup nobody has restored is a backup nobody knows works.
Do this: run terraform state pull and save the output to a file. Create a scratch workspace, push the saved state into it, and run terraform plan. Expected: the plan is empty, because the restored state matches the real cloud. If the plan shows changes, the backup is stale, and that is worth knowing now.
Read the flags: terraform state pull, terraform state push, and terraform workspace new
Pull downloads the state. Push uploads a state file. Together they are the manual backup and restore path, and the restore drill proves the backup works.
terraform state pull: This prints the current state as JSON. With no redirect it prints to the screen. It never changes anything.> state-backup.json: The greater-than sign redirects the output into a file. Without it, the state prints on screen and is gone when the terminal scrolls. The file is the backup.terraform workspace new restore-drill: This creates a fresh workspace to practise the restore in. A drill in a throwaway workspace cannot hurt prod state.terraform state push state-backup.json: This replaces the current state with the file. It is a loaded command. A push of the wrong file overwrites good state, so I only run it in a drill or a real recovery.Try it: run state pull into a file, create the restore-drill workspace, push the file back, and run terraform plan. Expect no changes, because the restored state matches the config.
terraform state pull > state-backup.json
terraform workspace new restore-drill
terraform state push state-backup.json
terraform plan
What the snippet means: pull copies the state out, push puts it back. The drill tests the whole path, backup to restore to empty plan, with nothing at risk.
The exit code is the drift alarm. A scheduled plan that reports drift needs a machine-readable answer, not a human reading the output. The detailed exit code gives it one. Zero means no changes. Two means the cloud moved. One means the plan itself failed.
Do this: run terraform plan -detailed-exitcode on a clean config and print the exit code. Expected: zero. Change a variable default, run it again, and expect two. That two is the signal a scheduled job can page on.
terraform plan -detailed-exitcode -out=tfplan; echo "exit code: $?"
What the snippet means: the exit code turns a plan into a check. Zero is quiet. Two is drift, and the job that runs this on a schedule can alert on it without parsing text.
Try it: Master
I own state for many teams. What proves my backup works before I ever need it for real?
Now do it: Run the drill above: pull the state to a file, push it into a restore-drill workspace, and expect an empty plan.
A remote backend does three jobs at once. It stores the state where the team can reach it. It versions every write, so a bad state can be rolled back to the one before it. And it locks, so two applies cannot write at the same time. The common pattern is an object store for the state and a small table for the lock. The state is encrypted at rest, and only the pipeline role may write it.
Workspaces and folders answer the same question two ways. A workspace keeps one set of files and swaps the state file per environment: dev, staging, prod. It works when the environments are nearly identical. Folders keep one folder per environment, each with its own files and its own state. It works when the environments differ in shape, not just in size. I use workspaces for small differences and folders for real ones. Mixing the two is how a dev variable ends up in prod.
Import is how hand-built resources join the files. You write the resource block, you import the real resource by its id, and the next plan shows the diff between the two. The diff is usually large, because the files do not yet describe the resource fully. You close the gap by editing the files until the plan is empty. Import never changes the cloud. It only changes the memory.
State surgery is for refactors. When the files rename a resource, the state still holds the old name, and the next plan would destroy the old resource and create a new one. State mv renames it in the memory, and the plan goes quiet. State rm forgets a resource entirely: Terraform stops managing it, and the cloud keeps it. Both commands need a state backup first, because there is no undo button.
Force-unlock is the command I respect most in this lesson. A stuck lock usually means a crashed apply. It can also mean a live apply that is slow. Force-unlock while an apply is running corrupts the state, and a corrupt state is the one failure in this lesson that has no quick fix. Before I force anything, I check the backend for the lock holder, I ask the team, and I check the pipeline. Only then do I unlock.
At 10x the resources, the shape holds and the numbers grow. The plan is slower, because it reads more of the cloud. At 100x, the design changes: the state splits by service, drift checks run on a schedule, and policy checks reject dangerous plans before they apply. The state file never stops being the memory. It just gets more owners.
terraform {
backend "s3" {
bucket = "tf-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "tf-locks"
}
}
What the snippet means: the state lives in an object store, under one key per environment, with a lock table beside it. Every write is versioned, so the previous state can be restored. Two applies cannot write at once, because the lock table says who is applying. Those seven lines are the whole state policy for this environment.
Write an import block and validate it. Then prove the point the lesson makes about the state being the memory: remove a resource from the state, watch the plan offer to create it, and bring it back from the backup.
import {
to = null_resource.box
id = "5553786609929720779"
}terraform validate
terraform state rm null_resource.box
terraform plan -detailed-exitcode; echo $?
cp terraform.tfstate.20261120.bak terraform.tfstate
terraform plan -detailed-exitcode; echo $? What you should see: validate prints the success line. The plan after the rm exits 2. The plan after the restore exits 0.
Read the flags: terraform state rm
State rm makes Terraform forget a resource. The cloud object keeps running. Forgetting is not deleting, and that difference is the whole point of the command.
terraform state rm: This deletes one address from the state file. Terraform stops managing it. The real object is untouched. I use it when a resource leaves Terraform management on purpose.null_resource.box (the address): This names the exact resource to forget. A wrong address forgets the wrong resource, so I copy it from state list first.terraform plan -detailed-exitcode (after rm): The plan after a rm shows the resource as new, because Terraform no longer knows it exists. That is expected. If I apply that plan, Terraform builds a second copy, so I read the plan before I touch apply.Try it: run terraform state rm on a scratch resource, then run terraform plan -detailed-exitcode. Expect exit code 2 and the resource shown as new. Restore the backup copy of the state file after.
It passes when: the restore brings the plan back to exit 0. Two honest notes. First, the id in the import block is a placeholder: read the real one from your own state file. Second, the local providers in this drill do not support import, so a live import needs a real provider. The backup restore is the drill that works locally, and it proves the same thing: the state is the memory, the cloud is the truth.
Force-unlock is the most dangerous command in this lesson, and the danger is not the command. It is skipping the check before it. A lock exists because a run is live, or because a run died. The fix is different for each, and the command is the same.
Do this: reproduce a stale lock by stopping a plan with Ctrl-C, then run terraform plan and read the lock error. Expected: the error names the lock ID, the run that holds it, and the time it started. Confirm no pipeline run is live, then clear it with the lock ID from the error. Run the plan again and expect it to pass. Write down the check you did, because the next person to clear a lock will copy it.
terraform force-unlock -force LOCK_ID
What the snippet means: the lock ID comes from the error, never from memory. The check before the command is the whole skill. The command itself is one line.
Read the flags: terraform force-unlock -force LOCK_ID
A crashed apply leaves its lock behind. The lock blocks every later run until someone clears it. Force-unlock clears it, and only it.
force-unlock: This removes the lock record from the backend. It never touches the cloud. I check no run is live first, because clearing a live lock lets two applies write one state.-force: This skips the yes or no prompt. Without it, Terraform asks first. In a script the prompt would hang, so -force is the only way the command works there.LOCK_ID: This is the lock ID from the error message. I copy it from the error, never from memory.Try it: start a plan, stop it with Ctrl-C, run terraform plan again, copy the lock ID from the error, and run force-unlock with it. Expect the next plan to pass.
A state file I owned was written by two applies at once. No lock was configured, because the backend had been set up quickly and locking was left off. Both applies finished. Both wrote the state. The second write erased the first apply's work from the record, while the cloud kept both sets of changes. The next plan offered to create resources that already existed.
Three things. First, locking is on in every backend, and a backend without locking is treated as broken. Second, applies run only from the pipeline, never from a laptop, so there is one writer by construction. Third, every state backend has versioning on, so a bad write can be rolled back to the state before it.
Replay the exact symptom from the incident: a plan that offers to create a resource that already exists. Remove the resource from the state, not from the cloud, and run plan.
terraform state rm null_resource.box
terraform plan -detailed-exitcode; echo $?
cp terraform.tfstate.20261120.bak terraform.tfstate
terraform plan -detailed-exitcode; echo $?
terraform state list What you should see: the first plan exits 2 and offers to create null_resource.box, which is still running. The second plan exits 0. terraform state list prints null_resource.box.
It passes when: the plan after the restore exits 0 and the state list shows the resource again. That is the whole incident in four commands: the state lost the resource, the plan believed the state, and the backup fixed the record. The fix that stops it happening again is locking on the backend, which is the S3 block from the Intermediate section.
Design state management for three environments (dev, staging, prod), two teams, and about 200 resources. One team owns the network, the other owns the application servers. Applies happen ten times a day. A bad apply must never touch prod, and a lost state file must be recoverable in under an hour.
Start with the budget from this lesson. Files 0, core 15, lock 5, state read 10, plan 15 plus half a millisecond per resource, apply about one millisecond per resource plus 35 per drifted resource, write-back 15, next run 5. For 200 resources with 2 drifted that is about 15 plus 100 plus 12 for the plan, 200 plus 70 for the apply, and 40 for the fixed hops, about 437 ms in this teaching example. The real cost is not the milliseconds, it is the blast radius: one state file for 200 resources means one bad apply can touch all of them, so split the state by team and by environment.
One state file buys simplicity and costs blast radius: every apply can touch everything. Many state files buy safety and cost composition: teams must look up each other's state to connect services. Remote state with locking buys safety and costs a backend to run. Workspaces buy one set of files and cost the risk that a dev variable leaks into prod. And the target is recovery, not perfection. A state file that can be restored in an hour beats a state file that was never lost.
Build state management for dev, staging, and prod in three folders. Each folder gets its own backend file and its own state path. Then run the recovery drill in dev: lose the state record, prove the plan notices, and restore it from the backup.
Steps:
main.tf.dev.for env in dev staging prod; do (cd $env && terraform init -backend=false && terraform validate && terraform fmt -check); done
cd dev
terraform apply -auto-approve
cp terraform.tfstate terraform.tfstate.bak
terraform state rm null_resource.box
terraform plan -detailed-exitcode; echo $?
cp terraform.tfstate.bak terraform.tfstate
terraform plan -detailed-exitcode; echo $? What you should see: Success! The configuration is valid. three times. The drill plan exits 2 after the rm and 0 after the restore.
It passes when: all three folders validate, fmt prints nothing, the drill plan exits 2 after the rm and 0 after the restore, and the prod folder never shares a state path with dev. That is the recovery target from the exercise, proven in minutes: a lost state file is recoverable, and the proof is a plan that goes quiet again.