Platform Engineering · Lesson 114

Grafana

Lesson 113 gave you the numbers. This lesson gives you the screen that shows them. Grafana does not store your metrics. It asks for them, draws them, and watches them. One checkout dashboard shows request rate, error rate, and p99 on one screen, all from the same Prometheus behind it. A variable at the top lets me pick the service. An alert on the error panel pages me when the line crosses 5 percent. I learned Grafana the way I learn most tools in this job. I built a dashboard with forty panels, watched it take a minute to load, and then read the queries until the queries confessed. This lesson walks one dashboard load from the browser to the alert, from a first panel to the deep end: dashboard as code, Loki and Tempo as data sources, and running it for more than one team without the teams stepping on each other.

Difficulty: Advanced · Know this first: Lesson 25

Run it as:

When it breaks, I check

One thing to remember

The Grafana map: browser, org, panels, three data sources

I start from scratch: one browser, one dashboard, one panel, one query against one data source. Then I add one piece at a time until I reach the full picture: variables feeding every panel, six panels firing at once across Prometheus, Loki, and Tempo, org/role checks deciding who may query what, and Grafana alerts with a panel link riding on the page.

6 PANELS20 VIEWERS30 s REFRESH
Browser 6 PANELS Grafana server and its Auth and org check Variables 20 VIEWERS Panels firing at once Data sources Alert notification 30 s REFRESH one dashboard load
loadqueryrenderalertRequestResponse
Step 1 of 7Map

Browser (opens one URL)

Browser (opens one URL) I start here. This is the simplest piece, and nothing else exists yet.

  • I can find Browser on the picture.
  • I can say what goes in here and what comes out.

The dashboard that became the incident: the refresh loop

From scratch to master. Starts from scratch: the simplest loop — one dashboard, six panels, one refresh, and the arithmetic of queries per second. Builds to the Master-level picture: the incident trip (40 panels x 80 viewers x 5-second refresh = 640 queries a second, the data source timing out) and the Master-level discipline — dashboards as load on the data source, sized and pruned like any other client.

40 PANELS80 VIEWERS640 QUERIES/S
Dashboard loads 40 PANELS Refresh every 30 seconds Panels grow to 40 Refresh stays at 5 seconds 80 viewers open the 80 VIEWERS 40 queries every 5 Prometheus query queue Master fix 640 QUERIES/S one dashboard load, re
normal loadoverloadtimeoutRequestResponse
Step 1 of 8Trip

Dashboard loads: 6 panels, 1 query each.

Dashboard loads: 6 panels, 1 query each. I start here. This is the simplest piece, and nothing else exists yet.

  • I can find Dashboard loads on the picture.
  • I can say what goes in here and what comes out.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one dashboard load, not a measurement from a live system, but the shape is honest: in most slow dashboards I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The slow dashboard takes almost three times as long without the data changing at all. Same metrics, same tool. Only the panel count and the time range changed, and the bill multiplied.

Try it: change the dashboard and watch the load bend

The budget bar above shows three fixed stories. This simulator lets you write your own. It models one dashboard load, with a panel count, a time range, and a crowd of readers on top. Move a slider and watch the queries per load, the load time, and the queries per second change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: panels multiply queries, the time range multiplies the work per query, and a crowd multiplies both.

6

What it changes: the queries per load and the render time. Queries per load equals panels, and render time grows with panels.

24

What it changes: the work per query. Query time grows with the range, because the data source reads more samples.

20

What it changes: the queries per second hitting the data source. Queries per second equals viewers times panels divided by the refresh interval.

30

What it changes: how often the dashboard re-asks. A shorter refresh means more queries per second for the same viewers and panels.

Try this: (1) Leave the defaults: 6 panels, 24 h range, 20 viewers, 30 s refresh. The load time is about 63 ms and the data source sees 4 queries per second. (2) Push the panels to 30, the range to 168 h, and the refresh to 5 s with 200 viewers. The load time jumps to about 735 ms and the data source sees 1,200 queries per second. Same metrics, same tool. Only the knobs changed. (3) Set everything back, then set the refresh to 300 s and watch the queries per second fall to 0.4. That is what a calm refresh costs in production: nothing, and it saves the data source.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Grafana looks like one tool. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: get a panel drawing one metric, then make the dashboard answer questions, then make the dashboard fast and honest, then run it for more than one team. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.

Basic - One panel, one query, one graph

Grafana at the basic level is a screen that asks a data source a question and draws the answer. A data source is a connection to Prometheus, Loki, Tempo, or another tool that holds data. A panel is one question and one drawing: a graph, a stat, a table. A dashboard is a page of panels. The query is written in the data source language, PromQL for Prometheus, LogQL for Loki. Nothing in Grafana stores the data. Grafana asks, the data source answers, and the panel draws.

Types:

  • Graph panel: a line over time, like request rate.
  • Stat panel: one big number, like current error rate.
  • Table panel: rows, like the slowest endpoints.
  • Gauge panel: a dial, like CPU use right now.
  • Data source: a connection to Prometheus, Loki, Tempo, or another store.

Uses: I use the basic layer when a team wants to see one service. One dashboard, a handful of panels, request rate, error rate, p99. The first win is not a feature. It is that the whole team looks at the same screen, and the same screen means the same story in the incident call.

Scale, failure modes, cost, trade-offs: Scale: this layer holds up to a few dozen dashboards without any special care. Failure modes: a panel with the wrong data source shows no data, and the fix is the data source dropdown, not the query; a dashboard that nobody owns rots, and the panels go stale one by one. Cost: one small Grafana server and its database, plus the queries the panels send. Trade-offs: you pay a query on every load and every refresh, so a dashboard nobody opens is cheap and a dashboard everyone leaves open is a load on the data source.

Intermediate - Variables, templating, and alerts from the panel

The intermediate layer is the dashboard that answers questions. A variable is a dropdown that feeds every panel: pick a service, and every query on the page uses it. A template variable can depend on another, so picking a cluster narrows the service list. An alert rule runs a panel query on a schedule and fires when the condition holds. A dashboard link ties a panel to the log or the trace behind it, so one click takes me from the graph to the log line.

Types:

  • Variables: dropdowns that feed every panel query.
  • Chained variables: one variable narrowing the next.
  • Grafana alerts: a panel query, a condition, and a schedule.
  • Panel links and data links: a click from a graph to the logs or the trace.
  • Annotations: markers on the graph for deploys and incidents.

Uses: I use this layer for the dashboard the team actually lives on. One dashboard per service, a variable for the environment, and the same page works for staging and production. The point of this layer is that a dashboard stops being a picture and starts being a tool: I pick a service, I see its health, and I click through to the logs when the graph says something is wrong.

Scale, failure modes, cost, trade-offs: Scale: this layer holds to a few hundred dashboards, but the variable cost grows, because every variable is a query that runs before the panels. Failure modes: a variable that returns a thousand values makes every panel wait; an alert query that differs from the panel query fires when the panel says it should not. Cost: the cost is queries, and the bill is paid by the data source on every load and every refresh. Trade-offs: a variable buys one dashboard for many services and costs a query before the panels can run. A Grafana alert buys an alert next to the graph and costs a Grafana that has to stay up.

Advanced - Dashboard as code, Loki and Tempo, and multi-tenancy

The advanced layer is the dashboard as a system. Dashboard as code means the dashboard JSON lives in a repo, reviewed like code, deployed like code, and rolled back like code. Loki is the log data source: logs by label, queried with LogQL, and linked from a metric panel so one click shows the log lines behind a spike. Tempo is the trace data source: a trace by id, linked from a log line so one click shows the trace behind the log. Multi-tenancy means orgs and teams, folders per team, and data source permissions per org, so one Grafana serves many teams without them seeing each other's dashboards.

Types:

  • Dashboard as code: JSON in a repo, provisioned on deploy, reviewed in a pull request.
  • Loki data source: logs by label, LogQL queries, log panels.
  • Tempo data source: traces by id, linked from logs and metrics.
  • Explore: the ad-hoc query view, for asking a question without building a dashboard.
  • Orgs, folders, and teams: the tenancy model that keeps teams apart.

Uses: I use this layer when the dashboards stop being a handful and start being a fleet. A repo that holds every dashboard, so a bad edit is a revert, not a rebuild by hand. A metric panel that links to the Loki logs behind the spike, and a log line that links to the Tempo trace behind the error. The point of this layer is that the three pillars stop living on three screens and start living on one, with clicks between them.

Scale, failure modes, cost, trade-offs: Scale: this layer holds to thousands of dashboards, but the provisioning has to be automated, because nobody can click a thousand dashboards into place. Failure modes: a dashboard as code merge that deletes a panel shows up as a missing panel, not an error; a Loki query with a bad label selector returns nothing and looks like no logs exist; a team in the wrong org sees an empty dashboard and files a bug against the data. Cost: the cost is operational: a repo, a pipeline, and a human who owns the provisioning. Trade-offs: dashboard as code buys review and rollback and costs a pipeline; multi-tenancy buys one Grafana for many teams and costs a permission check on every query.

A dashboard can be the most expensive query in the system. A template variable that lists every pod makes the dashboard run one query per pod, and a dashboard with twenty panels and a variable like that runs hundreds of queries on every load. I keep the variable lists short, I set the panel interval so the query asks for fewer points, and I cache the dashboard for a minute when the data does not change that fast. The check is the query inspector: I open it, I read the time each query took, and I fix the slowest one first. A dashboard that takes thirty seconds to load is a dashboard nobody opens during an incident, which is when it is needed most.

Master - Running Grafana like a platform: provisioning, performance, and the dashboard budget

The master layer is Grafana as a platform. Provisioning means dashboards, data sources, and alert rules all come from files, deployed by a pipeline, with no clicking in production. Performance means a dashboard budget: a cap on panels per dashboard, a cap on the time range, a recording rule behind every expensive query, and a query cache so a hundred viewers do not send a hundred identical queries. The master skill is not the panel. It is the budget: a number the team agrees on, a dashboard that shows the slowest dashboards, and a rule that a new panel needs a reason.

Types:

  • Provisioning: dashboards, data sources, and alerts from files, deployed by a pipeline.
  • Query caching: the same query answered once for many viewers.
  • Recording rules behind dashboards: precomputed series for the expensive panels.
  • Dashboard budget: a cap on panels, range, and refresh, with an alert when it is breached.
  • Grafana Enterprise and Mimir at a glance: the managed pieces for scale, and the long-term metric store behind them.

Uses: I use this layer when one Grafana serves a whole company. A platform team that provisions dashboards for fifty teams. A cache that keeps a hundred viewers from hammering Prometheus. A budget that stops a dashboard from growing to forty panels. And I use the slowest-dashboard list every month, because the slowest dashboard is usually the most important one, and the most important one is the one that has to stay fast.

Scale, failure modes, cost, trade-offs: Scale: this layer holds at thousands of dashboards and hundreds of viewers, because the cache absorbs the crowd and the recording rules absorb the expensive queries. Failure modes: a provisioning pipeline that overwrites a manual edit teaches the team to never edit by hand, and then someone edits by hand anyway; a cache that serves a stale answer makes the dashboard lie for a minute; a budget that nobody enforces is a number on a wiki. Cost: the cost is operational: a pipeline, a cache, and a team that owns the platform. Trade-offs: provisioning buys consistency and costs the freedom to click; caching buys scale and costs staleness; a dashboard budget buys survival and costs a conversation every time someone wants a new panel.

Alert state is where Grafana trips people up. An alert has three states that matter: pending, firing, and no data. Pending means the condition is true but the wait time has not passed, and the wait exists so a one-minute spike does not page anyone. Firing means the wait passed and the notification went out. No data means the query returned nothing, and the default is to treat that as nothing wrong, which is backwards for a dead service. I set no-data to alert on the alerts that watch availability, and I set it to keep quiet on the alerts that watch a metric that legitimately goes quiet. Then I test each alert by making its condition true on purpose and watching it move through all three states.

Go deeper

Dashboard as code is the first thing I reach for when a dashboard matters, and the first thing I check when a dashboard breaks. The JSON lives in a repo. A change is a pull request. A bad change is a revert. The trap is the drift: someone edits the dashboard by clicking in the UI, the repo still holds the old JSON, and the next deploy overwrites the click. I stop the drift by making the UI read-only in production, or by exporting the clicked change back into the repo the same day. A dashboard that lives in two places lives in neither, and the one that wins is the one the pipeline deploys.

Loki changes how I read logs, because Loki does not index the log line. It indexes the labels, and the line is searched at query time. That is the whole trade. A label with too many values is the same cardinality problem as Prometheus, and the fix is the same: few labels, low values, and the detail stays in the line. I link a metric panel to Loki with a data link, so a spike in the graph opens the log lines for the same service and the same minutes. The link carries the time range and the labels, and the on-call does not have to retype either at 2 AM.

Tempo changes how I read traces, because a trace is found by id, not by search. The log line carries the trace id, the metric panel links to it, and one click opens the trace. The trap is the sampling: Tempo only holds the traces that were kept, and a trace that was dropped is a link that goes nowhere. I keep the link anyway, because a link that works nine times out of ten is still faster than searching by hand, and I check the sampling rate before I blame the link.

Multi-tenancy is a permission model, not a folder structure. An org is the tenant. A folder is the drawer. A team is the group of people. The mistake is to build folders and call it tenancy, while every team can still query every data source. I check the data source permissions per org, because the dashboard permission decides who sees the panel and the data source permission decides who can run the query. A viewer in the wrong org should see nothing, and the test for that is to log in as that viewer and look.

At 10x the dashboards, the provisioning pipeline is the design. Nobody can click a thousand dashboards into place, and nobody can review a thousand clicks. The repo holds them, the pipeline deploys them, and the review is the pull request. At 100x, the design changes. The query cache becomes a real piece of infrastructure, because a hundred viewers asking the same question should not send a hundred queries. The recording rules move behind the dashboards, so the expensive panels read precomputed series. And the dashboard budget gets a number of its own, because at that scale a dashboard is a load on the data source, and a load nobody budgets for is a load nobody controls.

{
  "title": "Checkout health",
  "templating": {
    "list": [
      {"name": "service", "type": "query", "datasource": "Prometheus",
       "query": "label_values(http_requests_total, service)"}
    ]
  },
  "panels": [
    {
      "title": "Error rate",
      "datasource": "Prometheus",
      "targets": [
        {"expr": "sum by (service) (rate(http_requests_total{service=\"$service\",status=~\"5..\"}[5m])) / sum by (service) (rate(http_requests_total{service=\"$service\"}[5m]))"}
      ],
      "links": [
        {"title": "See logs", "url": "/explore?left={\"datasource\":\"Loki\",\"queries\":[{\"expr\":\"{service=\\\"$service\\\"} |= \\\"error\\\"\"}]}"}
      ]
    }
  ],
  "alert": {
    "name": "Checkout error rate high",
    "condition": "error rate > 0.05 for 5m",
    "contactPoint": "pagerduty-checkout"
  }
}

Four pieces, four jobs, one dashboard. The variable queries Prometheus for the list of services, so the dropdown always matches what exists. The panel runs one PromQL query for the error rate of the chosen service, and the data link carries the service name into Loki, so one click shows the error log lines for the same service. The alert watches the same error rate and pages the checkout contact point when it stays over 5 percent for 5 minutes. Read them together and you can see the whole design: one variable feeds every panel, one query feeds the panel and the alert, and one link ties the metric to the log. Change one of them without reading the other two and you have changed the dashboard in a way you did not intend.

Dashboards belong in version control like any other config. I keep the dashboard JSON in the repo, and a change goes through a pull request with a screenshot of the before and after. The reason is the 2 AM edit: someone fixes a panel during an incident, the fix is right, and nobody remembers what changed, so the next deploy overwrites it. With the JSON in the repo, the change is a commit with a message, and the overwrite cannot happen silently. I also keep one dashboard per service that answers the same four questions: is it up, is it fast, is it erroring, and is it saturated. Any panel that answers a different question goes on a second dashboard.

rules = 120
eval_every_s = 60
evals_per_min = rules * 60 / eval_every_s
print(f"{rules} rules every {eval_every_s}s = {evals_per_min:.0f} evaluations per minute")
print("each evaluation runs one query; double the rules and the query load doubles")

Expected output: 120 rules evaluated every 60 seconds is 120 evaluations a minute, one query each. The math is linear, which is why it is easy to miss: every new rule adds one query a minute forever, and the alerting load grows with the rule count even when nothing is firing.

One time it broke

This one stayed with me because the dashboard was the incident. The page was slow, the data source was fine, and the on-call blamed the wrong tool for an hour.

What broke

A checkout dashboard grew from 6 panels to 40 over a year, one panel at a time, each added for a good reason. Nobody removed a panel. The refresh stayed at 5 seconds, because that was the default when the dashboard was small. One morning, with 80 people watching the dashboard during a deploy, the page took a minute to load and the data source started timing out. The panels were fine one at a time. Together they were 40 queries every 5 seconds from 80 viewers, which is 640 queries a second against a Prometheus that was sized for 40. The dashboard that was supposed to show the deploy became the thing that slowed the tool that shows the deploy.

Timeline

  • 09:00:A deploy starts. Eighty people open the checkout dashboard to watch it.
  • 09:02:The dashboard takes a minute to load. The data source starts timing out on queries.
  • 09:15:On-call blames Prometheus. Prometheus is healthy, but its query queue is full of dashboard queries.
  • 09:30:On-call counts the panels: 40. Counts the viewers: 80. Counts the refresh: 5 seconds. The math is 640 queries a second.
  • 09:40:The refresh is changed to 60 seconds and the dashboard is split into three smaller ones. The load falls off a cliff.
  • 10:00:Prometheus recovers. The deploy finishes. The dashboard loads in seconds.
  • Next day:A dashboard budget is set: 12 panels max, 30 second refresh minimum, and a recording rule behind any panel over a day of data.

What changed after

Three things. First, a dashboard budget now caps panels at 12 and the refresh at 30 seconds minimum, and the slowest dashboards are listed every month. Second, the expensive panels read recording rules, so the dashboard asks for a precomputed series instead of a week of raw data. Third, the deploy dashboard is a separate small page with 6 panels, because the page everyone opens during a deploy is the page that has to stay fast when everyone opens it. The dashboard grew one panel at a time, and no single panel was the problem. The lesson is that a dashboard is a load on its data source, and a load that grows one panel at a time is a load nobody notices until it falls over.

If you remember nothing else

Design exercise

Design the Grafana setup for a platform with 30 services and 5 teams. Each team needs a dashboard per service, one shared overview per team, and an alert that pages when the error rate stays over 5 percent for 5 minutes. The dashboards must live in a repo, the logs must be one click from the metrics, and one team must never see another team's dashboards. Sketch the orgs, the folders, the variables, the data sources, and the provisioning, and say what you would measure.

Clarifying questions to ask

  • How many panels per dashboard, and what is the refresh, and what does that cost the data source at peak viewers?
  • Which panels read raw queries and which read recording rules, and how do you decide?
  • How does a metric panel link to Loki, and what does the link carry: service, time range, or both?
  • How is tenancy enforced: orgs, folders, or data source permissions, and how do you test it?
  • Who owns the dashboard repo, and what happens when someone edits a dashboard by clicking in the UI?

Rough sizing

Start with the budget from this lesson. One dashboard with 6 panels at a 30 second refresh is 0.2 queries per second per viewer. Thirty services times 5 teams is 150 dashboards, but only a handful are open at once, so I size for 50 concurrent viewers, which is 10 queries per second at the default, and 100 queries per second if the refresh drops to 5 seconds during a deploy. That is the number the data source has to hold. The recording rules sit behind the overview dashboards, so the overview reads one series per service instead of a raw query per panel. I size the query cache for the overview, because a hundred viewers asking the same question should send one query, not a hundred.

Trade-offs to name

More panels buy more information on one screen and cost more queries per load. A shorter refresh buys fresher data and costs more queries per second. Dashboard as code buys review and rollback and costs a pipeline, and the UI has to be read-only in production or the drift wins. Multi-tenancy buys one Grafana for five teams and costs a permission check on every query. And the alert lives where the query runs: a Grafana alert is next to the panel, a Prometheus alert survives Grafana being down. I pick the one whose failure I can afford.

Next in this series
Lesson 115 — OpenTelemetry

This lesson opened the dashboard that reads the metrics, and learned that panels multiply queries. The next lesson follows the trace that ties the whole request together: OpenTelemetry, from a first span to a collector that decides which traces to keep.