Dev Tools · Monitoring

Monitoring Tools

I wrote these for the nights when a dashboard goes red and the query in the runbook is one long line. Format the PromQL, check the alert rule before it pages you, and fix the Alertmanager config before the next deploy.

Prometheus Query Formatter

Paste a long PromQL query. I break it into readable lines: functions on their own lines, label matchers listed one per line, operators split out. I also check your brackets, so a missing paren shows up here instead of at 2 AM.

Formatted query

      

PromQL Cheat Sheet

The queries I actually reach for on call. Every example is copyable. I wrote each one from how I use it, not from a manual.

Rates and increases

Counters only go up, so you almost always wrap them in rate() or increase() before you graph them.

rate(http_requests_total[5m])
increase(http_requests_total{status=~"5.."}[1h])
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Latency with histograms

Never average a histogram. Use histogram_quantile on the rate of the buckets.

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

Aggregation

sum by keeps the labels you care about and drops the rest. Without by, you get one number for everything.

sum by (job) (rate(http_requests_total[5m]))
topk(5, sum by (instance) (rate(http_requests_total[5m])))
count(up == 1)
avg by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Recording rule patterns

Record the expensive query once, then alert and graph on the recorded metric. This is how I keep dashboards fast.

job:http_requests:rate5m = sum by (job) (rate(http_requests_total[5m]))
job:http_errors:ratio5m = sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m]))
job:http_latency:p95_5m = histogram_quantile(0.95, sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))

Up and down

The fastest check when something pages: which targets are actually down right now.

up == 0
absent(up{job="api"})

Grafana Variable Generator

Fill in the form and I build the dashboard variable JSON for you. Paste it into your dashboard JSON under templating.list, or use it as the spec when you add the variable by hand.

Label values builds label_values(<metric>, <label>) for you. Custom takes a comma list like prod, staging, dev. Interval takes values like 1m,5m,1h.

Dashboard variable JSON

      

Alert Rule Validator

Paste a Prometheus rule file or a single rule. I check the alert name, the expression, the for duration, the labels, and the annotations. A bad rule fails here, not after it pages the wrong person.

Alertmanager Config Formatter

Paste an Alertmanager config. I pretty-print it with clean two-space indentation and check the route, the receivers, and the timeouts. This handles common YAML: maps, lists, nesting, and quoted strings. It is not a full YAML parser, so exotic syntax may not parse.

Formatted config