The bill arrives once a month and most months nobody opens it. Then one month someone does, and the number is 40 percent higher than last month. Traffic did not grow 40 percent. Nothing launched. The bill grew anyway. This lesson is the walk I do when that happens: tag everything, split the bill, shrink what is idle, commit to the floor I know I will use, move cold data to cheap storage, delete what does nothing, and set an alert so the next surprise comes the same day, not at month end.
Difficulty: Intermediate · Know this first: Lesson 35, Lesson 48
Every step in this lesson charges rent in dollars, not milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live bill. The shape is honest: in most cloud bills I have chased, one or two lines eat most of the money. Step through the lesson or switch the scenario and watch the bar change.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the compute cost, the monthly savings, and the budget total change. The numbers are a teaching example, not live measurements. The shape is the lesson: idle capacity is the most expensive kind.
How many instances the fleet runs. One on-demand instance costs $180 a month, so the current cost is instances times $180. At 40 instances the current cost is $7,200 a month.
How much of the fleet is actually working. Rightsizing targets 70 percent utilization, so the right-sized count is instances times utilization divided by 70, rounded up. At 40 instances and 35 percent, that is 20 instances.
Share of the rightsized fleet bought at the committed discount instead of the on-demand price. The discount is 30 percent off the covered share, so the final cost is the rightsized cost times one minus 0.30 times coverage over 100. At 30 percent coverage the final cost is $3,276 a month.
Try this: (1) Defaults: 40 instances, 35 percent utilization, 30 percent coverage. Final cost $3,276 a month, saves $3,924 a month. (2) Worst end: 100 instances, 10 percent utilization, 0 percent coverage. Current cost $18,000 a month, right-sized count 15, final cost $2,700 a month, saves $15,300 a month. Same workload, same code. Only the sizing changed. (3) Already tight: 40 instances, 70 percent utilization, 60 percent coverage. Right-sized count 40, final cost $5,904 a month, saves $1,296 a month. When utilization is already at target, rightsizing finds nothing and only the commitment helps.
Three questions. No score is saved anywhere — this is just you against the material.
Unit economics first. The total bill is the wrong number to watch. The right number is cost per unit of work: cost per order, cost per active user, cost per request. I track unit cost next to the total. When the total rises because orders rose, that is growth. When unit cost rises while orders stay flat, something is leaking. I alert on unit cost drift, not on the total. The total will fool you every time growth is good.
Cost anomaly detection second. I feed daily cost into the same alerting path as latency. A service that spends 20 percent more than its seven day average pages the owning team. The hard part is not the math. It is the routing. An anomaly alert that goes to a shared inbox is a bill with extra steps. Route it to the team that owns the tag, and give the alert the tag values so the person paged can see what moved without opening three dashboards.
Kubernetes cost allocation third, and this builds on Lesson 48. In a shared cluster, the bill arrives for the whole cluster and the teams share it. I allocate by namespace labels first, then split shared costs like the control plane by each namespace's share of CPU and memory requests. Requests, not actual use, because requests are what reserve the capacity. But I show actual use next to it. A namespace that requests ten cores and uses one is paying for nine idle cores, and that gap is the conversation. Idle cost inside a cluster is still real money. It just hides better.
Showback versus chargeback fourth. Showback first, chargeback later. Showback means each team sees its number every week. Chargeback means the money actually moves out of their budget. I only switch to chargeback when the numbers are trusted and the teams can act on them. Charging a team for a number they cannot explain or change is how you lose trust in the whole program.
At 10x the spend, the manual weekly review breaks. Tagging, tiering, and waste reports have to run automatically, in the deploy pipeline and on a schedule, or they do not run at all. At 100x, cost becomes an engineering constraint like latency: every design review names its monthly cost, every new service ships with a budget and an owner, and the cost dashboard sits next to the latency dashboard. The shape from this lesson holds. The enforcement moves from a person to a system.
{
"tag_policy": {
"required_tags": ["team", "service", "environment", "cost-center"],
"enforce_at": "deploy",
"on_missing_tag": "block"
},
"budget_alert": {
"service": "checkout-api",
"monthly_budget_usd": 5000,
"alert_at_percent": 80,
"anomaly_vs_7day_average_percent": 20,
"notify": "team-checkout-oncall"
}
}
What the snippet means: four tags are required on every resource, enforced at deploy time. A deploy with a missing tag is blocked, not warned. The checkout service gets a $5,000 monthly budget with an alert at 80 percent, and a separate anomaly alert when any day's spend runs 20 percent above its seven day average. Both alerts go to the owning team's on-call, not to a shared inbox.
A nightly snapshot job kept every snapshot it ever took. There was no expiry and no retention rule. The snapshots were small at first, so nobody noticed. Over six weeks the storage line on the bill tripled. The data was fine. The backups worked. We were paying hot storage prices to keep copies of copies nobody would ever restore.
Three things. First, every backup job ships with an expiry on day one, and the expiry is reviewed like any other config. Second, the weekly waste report lists snapshot count and oldest age per service, so the next leak shows up as a count, not as a dollar surprise. Third, the storage budget alert fires on a 20 percent rise over the seven day average, and it pages the owning team. The bill still arrives once a month. The surprises do not wait for it anymore.
You run a SaaS on one shared cluster. Three teams share it. The bill is $48,000 a month and finance wants it at $30,000 without slowing deploys. Design the cost controls.
Start with the map from step 3. In a typical $48,000 bill, compute is about $20,000, storage about $9,000, and waste about $6,000. Rightsizing compute from 35 percent to 70 percent utilization cuts it roughly in half, to about $10,000. Committing 60 percent of that steady floor at a 30 percent discount saves about another $1,800. Tiering cold storage saves about $3,000. Deleting the waste saves the full $6,000. That lands near $30,000, and none of it needs a rewrite. The exact numbers come from the real bill. The shape is what the exercise tests.
Rightsizing trades headroom for money, so keep the peak covered with on-demand capacity and prove it with the load test from Lesson 35. Committing trades flexibility for price, so commit only to the floor and keep the peak on demand. Tiering trades retrieval speed for money, so keep what you restore often on the fast tier. Chargeback trades team goodwill for accountability, so start with showback until the numbers are trusted. And cost is a constraint, not the goal. A $30,000 bill with a slow checkout is not a win.