I used to change infrastructure by clicking around in the console. It worked until the day I could not say what had changed, or who changed it, or how to change it back. This lesson is the fix: describe the infrastructure in code, review it like code, and let a machine apply it. One change, nine hops, from a file on my laptop to real resources in the cloud.
Difficulty: Intermediate · Know this first: Lesson 33, Lesson 30
Every hop charges rent in minutes, not milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy change the plan review is the longest hop, and that is fine, because it is the hop that earns the apply. Switch the scenario and watch the bar change. In the drift scenario the plan and the apply both grow, because the change now includes work nobody wrote. In the lock scenario the apply grows because most of it is waiting.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the apply hop, the drift check, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: size and drift decide how long a change takes and how far it can hurt.
How many resources Terraform manages. Each one adds about 0.05 minutes to the apply, because apply calls the cloud once per resource, in dependency order.
Share of resources that drifted from the code because someone changed them by hand. Each drifted resource adds about 0.3 minutes to the apply and 0.2 minutes to the drift check, because the apply has to revert it and the check has to report it.
Try this: (1) Leave the sliders at the defaults: 120 resources, 5 percent drift. Six resources drifted, apply about 10 minutes, drift check about 6 minutes, blast radius 18 resources, detection window about 4.3 hours, total about 43 minutes. (2) Push both to the worst end: 1000 resources, 50 percent drift. Five hundred drifted, apply about 202 minutes, drift check about 105 minutes, blast radius 512 resources, detection window about 29 hours, total about 334 minutes. Same code, same team. Only the size and the drift changed.
Three questions. No score is saved anywhere — this is just you against the material.
Remote state is the first thing I fix on any new setup. The state file lives in object storage, with versioning on, and a lock table next to it. Terraform takes the lock before it writes and releases it after. If a run crashes, the lock stays, and someone has to clear it by hand after checking no run is live. A stale lock is annoying. Two writers with no lock is a disaster. I have seen a team lose a state file to an overwrite. They rebuilt it by hand, resource by resource, for two days.
Workspaces and modules solve different problems, and mixing them up causes real pain. A module is a reusable piece of infrastructure, like a standard database with backups and alarms already wired in. I write it once and use it in every environment. A workspace is a separate state file for the same code, so dev and prod can share the files but keep their own state. I use modules for reuse and workspaces for environments. I do not use workspaces to manage twelve different products. That is twelve state files wearing one name, and nobody can tell them apart.
Policy as code is the review that never gets tired. A rule says no public storage buckets, no databases without encryption, no security group open to the whole internet. The check runs on the plan, before apply, and a violation stops the run. I write the rule once and it reviews every change, including the ones at 2 AM. The rule is not there to slow the team down. It is there so the team does not have to remember every rule on every change.
Real systems are never greenfield. Something always exists already, built by hand before the code existed. Terraform import brings it into state without recreating it. I write the code to match what exists, I import, I run plan, and I fix the diff until the plan is empty. That empty plan is the proof the code and the cloud agree. Only then do I change anything. Importing is slow, careful work. It is also the only honest way to adopt a system that is already running.
At 10x the resources, the shape holds and the apply gets slow. I split the state by team or by service, because one state file for 4,000 resources is one blast radius for 4,000 resources. Plan takes minutes, and a bad plan blocks everyone. At 100x, the apply itself becomes the bottleneck, so I split further and I apply in parallel where the dependencies allow. The review does not change. The plan still gets read. The only thing that scales is the discipline.
terraform {
backend "s3" {
bucket = "infra-state-prod"
key = "network/terraform.tfstate"
region = "us-east-1"
encrypt = true
dynamodb_table = "infra-state-locks"
}
}
# policy: no public storage bucket, checked on every plan
package infra.storage
deny contains msg if {
input.resource_changes[_].type == "aws_s3_bucket_public_access_block"
input.resource_changes[_].change.after.block_public_acls == false
msg := "Public access block must stay on for every bucket."
}
What the snippet means: the state file lives in an encrypted bucket with versioning, and a DynamoDB table holds the lock. No two applies write the state at the same time. The policy runs on the plan and denies any bucket that turns off its public access block. The check runs before apply, so the bad change never reaches the cloud.
A team kept its Terraform state files on laptops. Two engineers applied changes to the same shared service in the same hour, each from their own copy of the state. The second apply ran from a state file that was a week old. Terraform looked at the old state, decided a database did not exist, and created a second one. The app kept writing to the old database. New signups landed in the new, empty one. Nobody noticed for a day, because both databases answered and both looked healthy.
Three things. First, state moved to remote storage with locking, and local state files were deleted from every laptop. Second, applies run only from the pipeline. Nobody applies from a laptop, not even for a small fix. Third, a nightly drift check compares the state with the cloud and pages if they disagree. A stale state file is now impossible, because there is only one state file, and only one writer.
Design the infrastructure as code setup for a team of 12 engineers managing 3 environments, dev, staging, and prod, and about 400 resources. One shared network, one database per environment, and a rule that no change reaches prod without a review and a plan.
Start with the numbers from this lesson. At 400 resources and a clean apply of about 0.05 minutes per resource, a full apply is about 20 minutes plus the fixed cost, so I plan for 30 and alert at 45. Split the state by environment, so dev, staging, and prod each have their own state file, and split the network into its own state because it changes rarely and everything depends on it. That is four state files, not one. Each environment is one module used three times, with different inputs. Twelve engineers means twelve possible writers, so applies run only from the pipeline, and the pipeline takes the lock.
One state file is simple and has one blast radius for all 400 resources. Four state files are more moving parts, and a change that crosses two of them needs two applies in the right order. Modules buy reuse and cost flexibility: a module that fits all three environments is a module that fits none of them perfectly, so expect a few environment specific overrides. Reviewing every plan buys safety and costs speed, so I let the pipeline auto apply dev, require a review for staging, and require a review plus a second approver for prod. And the state file is the crown jewel: versioned, locked, backed up, and readable by nobody who does not need it.