System Design · Lesson 25

Observability: Logs, Metrics, Traces

A user says the checkout was slow. That is all I get. No screenshot, no error code, just slow. My job is to turn that one sentence into a cause I can fix, and I do it with three tools that answer three different questions. The log tells me what happened to that one request. The metric tells me how many requests were slow and how slow. The trace tells me where the slow request spent its time. This lesson follows one slow request through all three. You will see the log line it leaves, the metric it moves, and the trace that points at the database query that ate the time. And you will see the version where the trace was never kept, and I am left with logs alone at 2 AM. That is why sampling is a decision, not a default.

Difficulty: Intermediate · Know this first: Lesson 01, Lesson 02
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most slow requests I have chased, one hop eats almost the whole budget, and the observability hops themselves cost only a few milliseconds each. Step through the lesson or switch the scenario and watch the bar change. In the blind spot scenario, watch the trace hops drop to zero. The request gets 8 milliseconds faster, and the timeline disappears with them.

Turn the knobs and watch the bill and the blindness move

The budget above is one fixed trip. This box is yours. Push the request rate up, write more log lines, and cut the sampling rate to 1 percent. Then ask yourself the question I ask at 2 AM: when the slow request comes, will I have its trace? I learned this the slow way: a sampling rate you never did the math on is a sampling rate that hides the exact request you need.

How much traffic arrives. It drives the log volume, the traces kept, and the rollup hop.

How much each request writes. It drives the log hop time and the log volume.

What share of requests keep a trace. A lower rate saves storage and blinds you to rare problems.

Log lines / second2,500216,000,000 per day
Log volume / day54 GBabout $810 / month
Traces kept / second504,320,000 per day
Chance a slow request keeps its trace10%at least one of ten: 65%

Assumptions, stated plainly: an average structured log line is about 250 bytes, and log ingestion is priced at $0.50 per GB — a round teaching number, not a vendor quote. The budget bar always shows a sampled request, so moving the sampling slider changes how many traces are kept and the chance a slow request is captured — not the trace hops in the total.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is one request through three pillars. Production is the rest of it. Here is what I actually spend my time on once the logging, the metrics, and the traces are running.

Logs: structured, sampled, and short

I log in JSON, one line per event, with the same field names everywhere: request id, route, status, duration, user id only where I need it for debugging. Plain text logs with five different formats are five different problems at 2 AM. I never log secrets, tokens, or full request bodies. I log at info level in production and I keep debug logging behind a flag I can turn on for one service, for ten minutes, without a deploy. Debug logging left on after an incident is how a log bill doubles in a week. I also cap log volume per service. When a service starts writing ten times its normal lines, something is wrong with the service or with the logging, and I want to know before the bill arrives.

Metrics: histograms, not averages, and labels with a budget

I record latency in histograms, never as a single average, because the average hides the tail and the tail is where the users live. I alert on p99, not on the mean. I keep a cardinality budget: every label value has to be bounded. Route, status code, and region are fine. User id, request id, and error message are not labels. They go in logs and traces. When a new label is proposed, I ask how many values it can take. If the answer is unbounded, the answer is no. I also set an alert on the number of metric series itself. If the series count jumps tenfold in an hour, a label exploded, and I want to know before the dashboards stop loading.

Traces: head sampling is a guess, tail sampling is a decision

Head sampling decides at the start of the request, before anyone knows how it ends. It is cheap and simple, and it throws away the interesting requests at the same rate as the boring ones. Tail sampling decides at the end, after the request finishes, and it keeps what matters: the errors, the slow ones, the ones over a threshold. It costs more, because every span has to be held until the decision is made, but it is the only sampling that keeps the trace for the one request I actually need. I run tail sampling in production wherever I can. The config below is the shape of it: keep every error, keep every request slower than 500 milliseconds, keep 1 percent of the rest.

Exemplars: the bridge between metrics and traces

A histogram bucket tells me ten requests were slow. An exemplar is a trace id attached to one of those slow requests, sitting inside the metric itself. I click the slow bucket and I land on the trace. That link is the difference between knowing something is slow and knowing which request was slow. If your metrics system supports exemplars, turn them on. It costs almost nothing and it saves the step where I go hunting for a trace id by hand.

What changes at 10x

At ten times the traffic, the log bill is the first thing that breaks, because log volume grows with traffic and nobody budgets for it. I cut log lines per request, I drop debug logs, and I sample the routine success logs while keeping every error log. Metrics survive 10x better than logs, because a counter costs the same whether it counts ten requests or ten thousand. Traces need tail sampling at 10x, because keeping every trace is a storage bill I cannot defend. I also split retention: traces live for 7 days, metrics for 90, logs for 30, errors for longer. Not everything needs the same shelf life.

What changes at 100x

At a hundred times the traffic, I stop thinking about one service and start thinking about the pipeline. Logs, metrics, and traces each get their own ingest path, their own budget, and their own backpressure, because a flood in one pillar must not take down the other two. I put a collector between my services and the backends, so a backend outage does not block my services. I cap the bytes per service, and when a service exceeds its cap, its logs are sampled down automatically instead of filling the disk. And I test the observability system the way I test the product: I break a service on purpose and check that the alert fires, the trace exists, and the dashboard shows it. Observability I have never tested is observability I am guessing about.

Tail sampling config I actually use

This is a generic collector tail sampling policy, not tied to any one vendor:

# Tail sampling: decide AFTER the request finishes, so the
# slow and the failed requests are the ones that survive.
# Head sampling at 1 percent would throw them away blind.
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      # Keep every error. An error with no trace is a dead end.
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      # Keep every slow request. Slowness is the incident.
      - name: slow-requests
        type: latency
        latency: { threshold_ms: 500 }
      # Keep 1 percent of the rest. The healthy baseline,
      # for comparing against when something goes wrong.
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }

The three policies are the whole design in a few lines. Errors always survive. Slow requests always survive. The healthy 1 percent is the baseline I compare against. A rare error at 1 percent head sampling has a 1 percent chance of keeping a trace. With this policy it has a 100 percent chance, because the decision is made after the outcome is known.

One time it broke

This one stayed with me because the dashboards went blind in the middle of an incident. The system that was supposed to tell me what was wrong was the thing that broke.

What broke

A deploy added a user id label to the checkout latency histogram. The idea was reasonable. The team wanted latency per user. The effect was not reasonable. Every user became a new metric series. The series count went from thousands to millions in an hour. The metrics backend slowed down, then the dashboards started timing out, then the alerts built on those metrics stopped firing on time. In the middle of it, a real checkout problem started. Requests were timing out for one region. The dashboards that would have shown it were loading forever, because they were querying a metric with millions of series. The logs still worked, and the traces still worked, but the metric that would have told me how many users were affected was the one that was broken. I was debugging blind on the one pillar that answers how many.

Timeline

  • 14:05A deploy ships the new user id label on the checkout latency histogram. Nobody flags the cardinality. The label looks harmless in the diff.
  • 14:40The metric series count starts climbing. Thousands become hundreds of thousands. No alert fires, because there is no alert on series count yet.
  • 15:20Dashboards that query the checkout histogram start loading slowly. The team notices the dashboards first, not the metric, and assumes the dashboard is the problem.
  • 16:10Checkout requests start timing out for one region. The p99 alert should fire. It fires late, because the alert query is slow against millions of series.
  • 16:25I check the metrics backend and see the series count. A million series, one per user. The label is the cause. I roll back the deploy and the label goes with it.
  • 16:50The series count stops growing. The old series take time to age out, but the dashboards come back within the hour. The checkout timeouts turn out to be a database connection limit in that region, found from the traces once the dashboards worked again.
  • 18:00The region's connection limit is raised and the timeouts clear. The incident ends with two fixes, not one: the connection limit, and the label that hid it.

What changed after

Three things, and I still do all three. First, a cardinality budget. Every new metric label has to answer how many values it can take, and unbounded labels are rejected in review. User id, request id, and error text are not labels. They go in logs and traces. Second, an alert on the metric series count itself. If the count jumps tenfold in an hour, I get paged before the dashboards die. Third, a rule I say out loud in every review now: metrics answer how many, and they can only do that if they stay cheap. The moment a metric gets expensive, it stops being a metric and becomes a liability. I also added a load test for the observability pipeline itself, so a label explosion is caught in staging instead of in production at 4 PM.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real pipeline. Take a minute with it before you read my approach.

The prompt

You run a checkout service at 500 requests per second at peak. You have logs, metrics, and traces, but they were set up in a hurry. Logs cost too much, the dashboards are slow, and the last incident had no trace for the failing request. Redesign the observability for this service. What do you log, what do you measure, what do you trace, and what do you keep?

Clarifying questions to ask first

I would ask five things before I sized anything. What is the current log volume per day, and what share of it is debug logging that should not be in production? Which dashboards are slow, and which metric has the most series behind it? What is the trace sampling rate today, and is it head sampling or tail sampling? How long do you need each pillar kept: traces for a week, logs for a month, metrics for a quarter? And when the last incident happened, which question went unanswered: what happened, how many, or where did the time go?

Rough sizing, worked

500 requests per second is 43.2 million requests a day. At 5 log lines per request and 250 bytes a line, that is 54 GB of logs a day and about $810 a month at $0.50 a GB. I cut it to 3 lines per request and sample the routine success logs, which brings it under 30 GB a day. Metrics: one histogram for checkout latency with labels for route and status code only, plus counters for requests and errors. No user id label. That is dozens of series, not millions. Traces: tail sampling that keeps every error and every request over 500 milliseconds, plus 10 percent of the rest. At 500 requests a second, that keeps 50 traces a second at baseline, more during an incident, and the slow requests always survive. Retention: traces 7 days, logs 30 days, metrics 90 days. Alerts: p99 latency over 500 milliseconds for 5 minutes, error rate over 1 percent for 5 minutes, and a new alert on metric series count jumping tenfold in an hour.

Trade-offs to name

I spend the log budget on errors and on the requests I cannot explain, not on every routine success. I keep metrics cheap by keeping labels bounded, and I accept that metrics will never name the one request that hurt. I spend the trace budget on tail sampling, because head sampling at 1 percent hides the rare failures, and I accept that tail sampling costs more to run. I keep retention short for traces and long for metrics, because a metric line from three months ago is still useful and a trace from three months ago is not. And I test the whole thing with a drill: I break the checkout on purpose in staging and check that the alert fires, the trace exists, and the dashboard shows it.

Next in this series
Lesson 26 — Monolith vs Microservices
Coming next