Lesson 25 gave you three pillars and one slow checkout. This lesson gives you the tool that holds up the metrics pillar for most teams I know. Prometheus does not wait for your app to send it numbers. It walks up to your app on a schedule, asks for the numbers at a URL, and writes down what it finds. That pull is the whole design. One checkout service exposes how many requests it served and how slow they were. Prometheus scrapes it every 15 seconds. A query in PromQL turns those scrapes into a graph. A rule watches the graph and pages me when the slow requests pile up. I learned Prometheus the way I learn most tools in this job. I added a label with a user id in it, watched the series count explode overnight, and then read the storage until the storage confessed. This lesson walks one metric from the app to the page, from a first scrape to the deep end: recording rules, federation, cardinality, and running it twice so one dead server does not take the alerts with it.
Difficulty: Advanced · Know this first: Lesson 25
Every hop charges rent in milliseconds. The bar below is a worked example for one metric travelling from the app to the page, not a measurement from a live system, but the shape is honest: in most slow alerts I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The cardinality explosion makes the same metric cost five times as much without the app doing anything different in kind. Same code path, same scrape. Only the label changed, and the bill multiplied.
The budget bar above shows three fixed stories. This simulator lets you write your own. It models one Prometheus server scraping a fleet of services, with a label choice and a query load on top. Move a slider and watch the ingest rate, the storage per day, and the query time change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: series are the currency, the scrape interval sets the resolution, and one bad label spends the whole budget.
More series means more samples per scrape, more memory in the head block, and more series for every query to read.
Samples per second equals active series divided by the scrape interval. A shorter interval sees outages sooner and writes more samples.
Series per metric equals the product of the label values, so one more label multiplies the series, it does not add to them.
Evaluation cost equals rules times 0.2 ms per rule, and every rule reads the series its query touches.
Try this: (1) Leave the defaults: 50,000 series, 15 s interval, 10 label values, 50 rules. The total is about 55 ms. (2) Push the series to 1,000,000 and the interval to 5 s. The storage jumps to about 25.92 GB per day. Same app, same metrics. Only the knobs changed. (3) Set everything back, then set the label values to 500 and watch the warning fire. That is what one user-id label costs in production.
Three questions. No score is saved anywhere — this is just you against the material.
Prometheus looks like one tool. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: get a scrape working and read one metric, then write queries that answer real questions, then make the answers fast and the alerts honest, then run it at a scale where one server is not enough. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.
Definition: Prometheus at the basic level is a server that pulls numbers from your app on a schedule and stores them as time series. The app exposes a /metrics page. Prometheus scrapes it every 15 seconds. Each line on the page is a metric name, some labels, and a number. A counter counts up. A gauge goes up and down. A histogram counts observations into buckets. That is the whole data model, and everything else in this lesson is built on it.
Types:
Uses: I use the basic layer when a team has a few services and wants to see what they are doing. One scrape per service. One dashboard with request rate and error rate. The first win is not a feature. It is that a number I can graph is a number I can argue about, and a number I can argue about is a number I can fix.
Scale, failure modes, cost, trade-offs: Scale: this layer holds up to a few dozen services and tens of thousands of series without any special care. Failure modes: a target that never gets scraped is invisible, so a wrong service-discovery label is the most common basic failure; an app that exposes a huge /metrics page makes every scrape slow. Cost: one small server, a little disk, and a scrape every 15 seconds per target. Trade-offs: you pay the pull on every target whether anyone reads the metric or not, so a metric nobody graphs is still a scrape, a series, and disk.
Definition: The intermediate layer is the query language and the rules. PromQL reads series, filters them by label, and does math over time: rate turns a counter into requests per second, histogram_quantile turns buckets into a p99, and aggregation by label turns a thousand series into one line. A recording rule runs a query on a schedule and stores the answer as a new metric. An alert rule runs a query, waits to be sure, and fires when the condition holds.
Types:
Uses: I use this layer for the questions on-call actually asks. How many requests per second, and how many of them failed. What is the p99, not the average. Is the error rate over the line, and has it stayed there long enough to be real. The point of this layer is that a metric stops being a number and starts being an answer.
Scale, failure modes, cost, trade-offs: Scale: this layer holds to a few hundred services, but the query cost grows with the series count, because every query reads the series it touches. Failure modes: a query with no label filter reads every series for a metric, and a recording rule that references a renamed metric writes nothing and shows a gap. A wait of zero pages on noise. Cost: the cost is query time and rule evaluation, both paid on every dashboard load and every evaluation tick. Trade-offs: a recording rule trades storage for speed, and an alert wait trades speed for honesty. There is no setting that buys both.
Definition: The advanced layer is the cost of the data model. Cardinality is the number of series, and it is the number that decides whether Prometheus survives. One label with a thousand values multiplies the series by a thousand. Retention decides how long the disk keeps it. Federation lets one Prometheus pull aggregated series from another, so a global server can see across clusters. High availability means running two Prometheus servers that scrape the same targets, with a deduplication layer in front of the alerts.
Types:
Uses: I use this layer when the series count stops being a number and starts being a bill. A label audit that finds the user-id label. A federation setup so one dashboard can see two clusters. A second Prometheus so one dead server does not take the alerts with it. The point of this layer is that Prometheus stops being a tool you install and starts being a system you operate.
Scale, failure modes, cost, trade-offs: Scale: this layer is where the single server stops being enough, usually around hundreds of thousands of series or a retention longer than a few weeks. Failure modes: a bad label explodes the series count overnight, and the disk fills before anyone notices; a federation that pulls raw series instead of aggregated ones copies the explosion to the global server; two servers with no dedup page twice for one outage. Cost: the cost is disk, memory, and a human who watches the series count like a budget. Trade-offs: federation buys a global view and costs a second hop on every query; remote write buys long retention and costs a dependency on the remote store; HA buys survival and costs deduplication work.
Definition: The master layer is Prometheus as a platform. Thanos and Cortex are the two names I meet most. Thanos puts a sidecar next to each Prometheus, uploads blocks to object storage, and gives one global query that deduplicates the two HA copies. Cortex takes remote write and stores the series itself, so the Prometheus servers stay small and the long-term store does the heavy lifting. Both exist because one Prometheus server has a ceiling, and the ceiling is the series count times the retention. The master skill is not the tool. It is the series budget: a number the team agrees on, a dashboard that shows it, and a rule that a new label needs a reason.
Types:
Uses: I use this layer when one Prometheus per cluster is not a design anymore. A platform team that runs Prometheus for fifty teams. A global view that has to survive one cluster going down. A retention of a year for capacity planning, at a resolution nobody needs to be 15 seconds. And I use the series budget every time a team asks for a new label, because a budget is the only thing that makes a label a decision instead of an accident.
Scale, failure modes, cost, trade-offs: Scale: this layer holds at millions of series, because the storage moves to object storage and the query moves to a layer that can fan out. Failure modes: a Thanos sidecar that stops uploading leaves a gap in the long-term store that nobody sees until the quarter review; a Cortex tenant that blows its budget takes the shared store down with it; a downsampled series that gets queried as if it were raw gives a p99 that is smoother than the truth. Cost: the cost is operational: object storage, a query layer, and a team that owns the platform. Trade-offs: Thanos buys a global view over servers you already run and costs a sidecar and a query layer; Cortex buys scale and costs a second system to operate; a series budget buys survival and costs a conversation every time someone wants a new label.
Recording rules are the first thing I reach for when a dashboard gets slow, and the first thing I check when a dashboard goes stale. A recording rule runs a query on a schedule and stores the answer. The dashboard reads the answer instead of the raw data. The trap is the schedule. A rule that runs every minute stores a point every minute, and a dashboard that asks for a 5 minute rate over those points is reading a precomputed answer at a coarser resolution than the raw data. That is fine when I know it. It is a wrong p99 when I do not. I name recording rules so the name says what they are: a recorded rate, not the raw counter. And I alert on a recording rule going stale, because a stale recording rule is a dashboard that lies quietly.
Cardinality is not a number. It is a multiplication. A metric with three labels, each with ten values, is a thousand series. Add one label with a hundred values and it is a hundred thousand. The fix is not a bigger disk. The fix is to decide, before the label exists, whether anyone will ever filter by it. A user id is almost never a label. An endpoint is often a label. A version is usually a label. I audit labels the way I audit spend: once a month, top series by count, and every label that grew gets a question. The relabelling config is where I enforce the answer, dropping the label at scrape time so the series is never created.
Federation has one honest use and one dishonest one. The honest use is a global server that pulls aggregated series from each cluster: the request rate per service, the error rate per service, nothing per pod, nothing per user. The dishonest use is pulling raw series across clusters because it is easier than aggregating. That copies the cardinality problem to the global server and doubles the scrape load on the cluster servers. I federate aggregates only, and I keep the federation interval longer than the scrape interval, because the global server does not need 15 second resolution. It needs a view that survives one cluster going down.
High availability in Prometheus is two servers and a lie you have to manage. Two servers scrape the same targets, so one dead server costs nothing. But two servers also fire the same alert twice, and two servers also store the same samples twice. The dedup has to live somewhere, and the somewhere is usually Thanos or Cortex, or Alertmanager grouping by alert name and labels. I do not run two servers and hope the pages sort themselves out. I run two servers, one Alertmanager cluster, and a grouping rule that makes two copies of one alert into one page. And I test the failover by killing one server on a quiet afternoon, not by waiting for it to die at 2 AM.
At 10x the services, the series count is the design. I stop asking how many services we have and start asking how many series each service is allowed. A budget per team, a dashboard that shows it, and an alert at 80 percent. At 100x, the design changes. The Prometheus servers become collectors that scrape and forward, and the storage moves to a system built for it, Thanos or Cortex, with object storage behind it. The query moves to a global layer. And the retention splits: 15 days at full resolution for debugging, a year at 5 minute resolution for capacity planning. Nobody needs a 15 second sample from March. They need a trend, and a trend is cheaper than a sample.
Three pieces, three jobs, one metric trip. The scrape config pulls checkout every 15 seconds and drops the user_id label at scrape time, so the series is never created and the cardinality explosion never starts. The recording rule precomputes the 5 minute error rate per service once a minute, so the dashboard reads one series instead of running the raw query over a day of data. The alert rule watches that precomputed series, fires when it stays over 5 percent for 5 minutes, and hands Alertmanager a page with a severity label. Read them together and you can see the whole design: drop the bad label early, precompute the expensive answer, and wait before you page. Change one of them without reading the other two and you have changed the trip in a way you did not intend.
# Drop the user_id label at scrape time, before the series is created.
# Without this, one series per user lands in the TSDB.
scrape_configs:
- job_name: checkout
scrape_interval: 15s
static_configs:
- targets: ['checkout:8080']
metric_relabel_configs:
- source_labels: [user_id]
action: labeldrop
# Recording rule: precompute the 5m error rate per service.
groups:
- name: checkout.rules
interval: 1m
rules:
- record: job:http_error_rate_5m
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))
- alert: CheckoutErrorRateHigh
expr: job:http_error_rate_5m{job="checkout"} > 0.05
for: 5m
labels:
severity: page
annotations:
summary: "Checkout error rate over 5% for 5 minutes"
This one stayed with me because the dashboard was fast and the disk was full at the same time. The label was small, the series were many, and nobody noticed until the store stopped writing.
A team added a user id as a label on a request counter, so they could see errors per user. It worked in testing, where there were ten users. In production there were two hundred thousand. Every user became a new time series, and every scrape created series that had never existed before. The series count went from 50,000 to over 900,000 in a day. Ingest slowed, queries that used to take 25 ms took 180 ms, and the disk filled in three days instead of three weeks. The alert for disk space fired, but the alert for the series count did not exist, so the first sign was a full disk. The metric that was supposed to help debug errors per user became the thing that broke the tool that shows errors.
Three things. First, a series-count alert now fires at 80 percent of the budget, so the next explosion is caught in an hour, not in three days. Second, every new label needs a reason in the deploy review, and a user id is never a label. The review asks one question: will anyone ever filter by this label, and what happens to the series count if the answer is yes for every user. Third, the relabelling config drops known-bad labels at scrape time by default, so a bad label is a dropped series, not a full disk. The deploy that broke this was a one-line change. The lesson is that a label is not a field. It is a multiplication, and a multiplication with two hundred thousand users is a number nobody can store.
Design the Prometheus setup for a platform with 40 services across 2 clusters. Each service exposes request counts, durations, and error counts. The team wants a dashboard per service, an alert that pages when the error rate stays over 5 percent for 5 minutes, and 30 days of retention for capacity planning. The global view must survive one cluster going down. Sketch the scrape config, the recording rules, the alert rules, the federation, and the HA setup, and say what you would measure.
Start with the budget from this lesson. One service at 50,000 series and a 15 second interval is 3,333 samples per second and about 0.43 GB per day. Forty services at the same size is about 133,000 samples per second and about 17 GB per day, and 30 days of that is about 510 GB before compaction. That is the number the disk has to hold. The global server federates aggregates only, one series per service per metric, so it holds a few hundred series, not two million. The recording rule precomputes the error rate once a minute, so the dashboard reads one series per service and the alert reads the same one. I size the HA pair for the same load each, because one server has to carry the whole fleet when the other is down, and I alert at 80 percent of the series budget.
A shorter scrape interval buys faster alerts and costs more samples, more disk, and more ingest. Federation buys a global view and costs a second hop on every global query. Two servers buy survival and cost deduplication, and the dedup has to live somewhere real, not in hope. A recording rule buys a fast dashboard and costs a series that has to be kept fresh. And the retention split is a trade: 15 days at full resolution for debugging, 30 days at 5 minute resolution for capacity planning. Nobody needs a 15 second sample from a month ago. They need a trend, and a trend is cheaper.