Platform Engineering · Lesson 182

Maths and Statistics for Data Work

Every on-call decision I make starts with numbers, and most of them start with bad numbers. A mean that hides one slow request. A test that calls noise a win. A forecast quoted as a fact. This lesson is the maths underneath those decisions, worked by hand on one week of latencies. Linear algebra, calculus, probability, hypothesis testing, regression, time series, and A/B testing. No theory for its own sake. Each tool answers one question I actually ask at work, and each one is run in Python before it goes on the page.

Difficulty: Intermediate · Know this first: Lesson 16, Lesson 19, Lesson 12

What you will build

By the end of this lesson you will have worked one week of latencies by hand, in Python, and you will have the scripts to do it again on your own data. First, a lab script that computes the mean, the median, the spread, the percentiles, a dot product, a regression line, a Bayes update, and a forecast, all on the same 20 numbers. Second, a real A/B test on two connection pools, with the interval and the p value, and a verdict you can defend. Third, a simulator that shows what sample size, outliers, spread, and trend do to the numbers you quote. This lesson is the maths prerequisite for Lesson 184, Classic Machine Learning. Lesson 184 builds on every tool here, so the regression line and the test in this lesson come back there as the training step.

What you need: Python 3 with numpy and scipy installed, with pip install numpy scipy. Nothing else. No cloud account, no live system, no dashboard. The numbers in this lesson are a teaching example, worked on 20 latencies written down in the lab. Your numbers will differ. The maths will not.

The story: one week, twenty numbers, seven tools

I keep a list of twenty request latencies from one week. Most sit between 142 ms and 245 ms. One took 580 ms. That list is the whole lesson. Every tool in it answers one question about that list, and the table below is the map. Read it top to bottom, then walk the diagram hop by hop.

ToolThe question it answersThe answer on this week
Mean and medianWhere is the centre.Mean 215.80 ms, median 199.00 ms. The gap is one slow request.
Variance and standard deviationHow far do the numbers stray.Sample std 89.90 ms, or 27.83 ms without the 580 ms request.
Percentiles and shapeWhat does the slow tail look like.p90 236.90 ms, p95 261.75 ms, interquartile range 41.25 ms.
Linear algebraDo two rows of numbers move together.Dot product 66000.0, cosine similarity 0.9915 for CPU and latency.
CalculusHow fast is it changing.Trend slope 8.82 ms per day on the daily p95.
Probability and BayesGiven a slow request, where do I look first.P(deploy given slow) is 0.60. Look at the deploy first.
Hypothesis testingIs the gap real, or is it noise.Mean against 200 ms: t is 0.79, p is 0.44. No evidence.
RegressionHow much does one thing move the other.Latency is 2.13 times CPU plus 76.00. At 100 percent CPU: 289.33 ms.
Time series and forecastWhat does next week look like.Day 8 p95 forecasts 270.14 ms, day 14 forecasts 323.07 ms.
A/B testingDid the change help, and by how much.New pool is 17.30 ms faster, interval -21.33 to -13.27 ms, p is 4.61e-10.

Three habits run through the whole lesson. First, never quote a centre without a spread. A mean without a standard deviation is a claim, not a measurement. Second, never quote a forecast without an interval. A line is a guess with good formatting until the interval sits beside it. Third, never call a gap a win until the test says it is bigger than the noise. A 17.30 ms gap with an interval clear of zero is a win. The same gap with an interval that crosses zero is a shrug.

The trap in this lesson is the word regression. The course already uses it for regression testing, which is re-running old tests after a change. That is not this lesson. Statistical regression fits a line through points. Time series already appears in the course as a data shape, the shape of a metric over time. That is not this lesson either. Here a time series is a list with dates on it, and the work is the trend and the forecast. A/B testing already appears as comparing two prompts or two model answers. Here it is a method: split the traffic, measure the gap, test the gap against the noise, and only then ship.

Do this now, on paper

Write down ten latencies from a system you watch. Any ten. Add them up, divide by ten, and write the mean. Sort them, take the middle two, average them, and write the median. If the mean and the median disagree by more than a few milliseconds, find the value pulling them apart. That value is the most interesting number on your list. The lab below does the same work in Python, on twenty numbers, with the spread and the percentiles beside them.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one week of latencies through the seven tools. These four paths are the same idea drawn four ways. One clean week, worked end to end. One week with four outliers, stopped at the spread. One budget trail, hop by hop, adding up to the 104 ms clean-week total. One small sample, where the numbers are the uncertainty itself. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the spread hop and never come back.

Path 1: the clean week

Outliers: one, named

Twenty latencies walk the tools in order. Centre, spread, shape, test, regression. The 580 ms request is named at the spread hop and carried honestly to the forecast, where the moving average keeps it from owning the answer.

Collect0 ms
Centre4 ms
Spread6 ms
Shape8 ms
Test18 ms
Regression22 ms

Not run yet.

Path 2: the outlier week

Forecast: skipped

Same week, four slow requests instead of one. The mean is pulled up 13.33 ms and the spread jumps to 74.54 ms. The trip stops at the spread hop. There is no honest forecast to carry forward until the outliers are named, so the regression and the forecast never run.

Collect0 ms
Centre4 ms
Spreadoutlier
Shapeskipped
Regressionskipped

Not run yet.

Path 3: the budget trail

Outliers: skipped

The clean week, priced hop by hop in work time. Each hop shows its share of the thinking. The trail adds up to 104 ms, the same total the budget bar shows for the clean week. The thinking hops, test and regression plus forecast, are most of it.

Sample0 ms
Centre + Spread10 ms
Shape8 ms
Vectors + Slope22 ms
Probability9 ms
Test18 ms
Regression + Forecast37 ms

Not run yet.

Path 4: the small sample

Certainty: skipped

The same week with only 12 samples. Each hop shows the uncertainty, not the time. The standard error is 5.77 ms, the interval is 11.32 ms wide on each side, and the smallest effect the test can see is 22.86 ms. The numbers are honest. The decision is not ready.

Samplen=12
Centremean 180
Spreadstd 20
TestSE 5.77
ForecastCI 11.32
VerdictMDE 22.86

Not run yet.

Where the time goes

Every hop charges rent in work time. The bar below is a worked example for one week of latencies, not a measurement from a live system, but the shape is honest. The thinking hops cost the most: the hypothesis test at 18 ms and the regression at 22 ms. The arithmetic hops are cheap. At the simulator defaults the clean week totals 104 ms. Move a simulator slider and watch the spread, test, regression, and forecast hops move with it. Switch the scenario and watch the test hop or the regression hop change instead.

Sample size or outliers: move the sliders and watch the numbers

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: a bigger sample shrinks the uncertainty, and outliers grow it faster than any sample can shrink it. You set how many requests are in the sample, how many of them are outliers 400 ms above the base, how wide the base spread is, and how fast the trend climbs. Move a slider and watch the mean, the standard deviation, the standard error, the interval, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: outliers own the mean, and the sample size owns the interval.

120 requests

How many latencies are in the sample. More requests shrink the standard error, because the error divides by the square root of n.

0 outliers

How many requests sit 400 ms above the base mean of 180 ms. Each one pulls the mean up and widens the spread.

20 ms

How wide the ordinary requests spread around the base mean, before any outlier. A noisier system needs a bigger sample to say the same thing.

2 ms/day

How fast the latency climbs each day. A steeper trend makes the regression hop work harder, and the forecast matters more.

Try this: (1) At the defaults, 120 requests, 0 outliers, base spread 20 ms, trend 2 ms a day, the mean is 180.00 ms and the standard deviation is 20.00 ms. The standard error is 1.83 ms, the 95 percent interval is 3.58 ms wide on each side, and the smallest effect the test can see is 7.23 ms. The budget total is 104 ms. (2) Add 4 outliers. The mean rises to 193.33 ms and the standard deviation jumps to 74.54 ms. The standard error is 6.80 ms, the interval is 13.34 ms wide, and the smallest detectable effect is 26.94 ms. The spread hop costs 14 ms, the test hop 26 ms, the regression hop 34 ms, and the budget total is 132 ms. (3) Go back to 0 outliers and drop to 12 requests. The mean is back to 180.00 ms and the spread is back to 20.00 ms, but the standard error is 5.77 ms, the interval is 11.32 ms wide, and the smallest detectable effect is 22.86 ms. The test hop costs 30 ms, the forecast hop 28 ms, and the budget total is 129 ms. The centre looks fine. The decision is not ready.

Formulas, in plain terms. Mean = 180 plus 400 times outliers over n. Variance = base spread squared plus the outlier share times one minus the outlier share times 400 squared. Standard deviation = the square root of the variance. Standard error = standard deviation over the square root of n. 95 percent interval half width = 1.96 times the standard error. Minimum detectable effect = 2.8 times the standard error times the square root of 2. Spread hop = 6 ms plus 2 ms per outlier. Test hop = 18 ms, plus 8 ms if there is any outlier, plus 12 ms if n is below 30. Regression hop = 18 ms plus 2 ms per unit of trend plus 3 ms per outlier. Forecast hop = 15 ms, plus 13 ms if n is below 30. Budget total = the ten hop times added up. The base mean of 180 ms and the outlier size of 400 ms are teaching figures. Your system will differ. The shape will not.

Lab: run the maths yourself

Do this now, on your own machine. You will run the lab script on twenty latencies, run a real A/B test on two connection pools, and break both on purpose to see what changes. The Python parts run on this machine and their outputs are shown exactly as they ran. No cloud account, no live system. Your numbers will differ when you use your own data. The maths will not.

Step 0: run the lab script on one week of latencies

Save this file as maths_lab.py. It works the twenty latencies from the story: mean, median, spread, percentiles, a dot product, a regression line, a Bayes update, a trend forecast, and a one-sample test against 200 ms. Run it and check the numbers against the story table.

"""Maths for one week of latencies, worked by hand with numpy.

The list is 20 request latencies in ms from one week. One of them,
580 ms, is a real slow request after a bad deploy, not a typo.
Every number below is computed here, not typed in by hand.
"""
import math

import numpy as np
from scipy import stats

lat = np.array([142, 158, 165, 171, 176, 180, 183, 188, 192, 197,
                201, 205, 210, 214, 219, 224, 230, 236, 245, 580],
               dtype=float)

print("n =", len(lat))
print("mean = %.2f ms" % lat.mean())
print("median = %.2f ms" % np.median(lat))
print("sample std = %.2f ms" % lat.std(ddof=1))
print("p90 = %.2f ms" % np.percentile(lat, 90))
print("p95 = %.2f ms" % np.percentile(lat, 95))
print("IQR = %.2f ms" % (np.percentile(lat, 75) - np.percentile(lat, 25)))
print("mean without the 580 = %.2f ms" % lat[:-1].mean())
print("median without the 580 = %.2f ms" % np.median(lat[:-1]))
print("std without the 580 = %.2f ms" % lat[:-1].std(ddof=1))

cpu = np.array([30., 45., 60., 75., 90.])
lat5 = np.array([150., 170., 190., 230., 280.])
print("dot(cpu, latency) = %.1f" % np.dot(cpu, lat5))
print("cosine(cpu, latency) = %.4f" % (np.dot(cpu, lat5)
      / (np.linalg.norm(cpu) * np.linalg.norm(lat5))))
slope, intercept = np.linalg.lstsq(
    np.vstack([cpu, np.ones(len(cpu))]).T, lat5, rcond=None)[0]
print("regression: latency = %.2f * cpu + %.2f" % (slope, intercept))
print("predicted latency at 100 percent cpu = %.2f ms" % (slope * 100 + intercept))

days = np.arange(1, 8)
p95 = np.array([210., 218., 225., 240., 232., 251., 268.])
trend, base = np.polyfit(days, p95, 1)
print("trend slope = %.2f ms per day" % trend)
print("forecast day 8 p95 = %.2f ms" % (trend * 8 + base))
print("forecast day 14 p95 = %.2f ms" % (trend * 14 + base))
ma3 = np.convolve(p95, np.ones(3) / 3, mode="valid")
print("last 3-day moving average = %.2f ms" % ma3[-1])

p_slow_given_deploy = 12 / 30
p_deploy = 30 / 100
p_slow = 20 / 100
print("P(deploy | slow) = %.2f" % (p_slow_given_deploy * p_deploy / p_slow))

se = lat.std(ddof=1) / math.sqrt(len(lat))
print("standard error of the mean = %.2f ms" % se)
print("95 percent CI half width = %.2f ms" % (1.96 * se))
t, p = stats.ttest_1samp(lat, 200)
print("one-sample t vs 200 ms: t = %.2f, p = %.4f" % (t, p))
python3 maths_lab.py

Expected output, exactly as it ran on this machine:

n = 20
mean = 215.80 ms
median = 199.00 ms
sample std = 89.90 ms
p90 = 236.90 ms
p95 = 261.75 ms
IQR = 41.25 ms
mean without the 580 = 196.63 ms
median without the 580 = 197.00 ms
std without the 580 = 27.83 ms
dot(cpu, latency) = 66000.0
cosine(cpu, latency) = 0.9915
regression: latency = 2.13 * cpu + 76.00
predicted latency at 100 percent cpu = 289.33 ms
trend slope = 8.82 ms per day
forecast day 8 p95 = 270.14 ms
forecast day 14 p95 = 323.07 ms
last 3-day moving average = 250.33 ms
P(deploy | slow) = 0.60
standard error of the mean = 20.10 ms
95 percent CI half width = 39.40 ms
one-sample t vs 200 ms: t = 0.79, p = 0.4416

Read the first eight lines twice. The mean is 215.80 ms and the median is 199.00 ms. The standard deviation is 89.90 ms with the slow request and 27.83 ms without it. That is the whole lesson in three numbers. The test at the end says t is 0.79 and p is 0.44 against 200 ms. The mean looks high, and the test says the data cannot prove it moved. One slow request is not a trend.

Now break it on purpose. Delete the 580 from the list and run it again. The mean falls to 196.63 ms, the median moves to 197.00 ms, and the standard deviation falls to 27.83 ms. The mean and the median agree now, because the outlier is gone. Change it back when you are done. The point of the break is not the new numbers. It is seeing one value own three statistics.

Step 1: run the A/B test on two connection pools

Save this file as ab_test.py. Group A is the old pool, group B is the new pool, twenty requests each. The script prints the means, the difference, the standard error of the difference, the 95 percent interval, the Welch t statistic, the p value, and the verdict at alpha 0.05.

"""A/B test by hand: did the new connection pool cut latency?

Group A is the old pool, group B is the new pool. Twenty requests
each, latencies in ms. The question is not whether B looks faster.
The question is whether the gap is bigger than the noise.
"""
import math

import numpy as np
from scipy import stats

a = np.array([180, 190, 175, 200, 185, 195, 178, 188, 182, 192,
              186, 198, 179, 189, 183, 194, 181, 196, 187, 193],
             dtype=float)
b = np.array([165, 175, 160, 180, 170, 178, 162, 172, 168, 176,
              164, 174, 166, 171, 169, 177, 163, 173, 167, 175],
             dtype=float)

print("A mean = %.2f ms, std = %.2f ms" % (a.mean(), a.std(ddof=1)))
print("B mean = %.2f ms, std = %.2f ms" % (b.mean(), b.std(ddof=1)))
diff = b.mean() - a.mean()
print("difference (B - A) = %.2f ms" % diff)
se = math.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
print("standard error of the difference = %.2f ms" % se)
print("95 percent CI for the difference = %.2f to %.2f ms"
      % (diff - 1.96 * se, diff + 1.96 * se))
t, p = stats.ttest_ind(b, a, equal_var=False)
print("Welch t = %.2f, p = %.2e" % (t, p))
print("verdict at alpha 0.05: " + ("reject the null, ship B"
      if p < 0.05 else "cannot reject the null, do not ship B"))
print("minimum detectable effect at this noise = %.2f ms"
      % (2.8 * se))
python3 ab_test.py

Expected output, exactly as it ran on this machine:

A mean = 187.55 ms, std = 7.16 ms
B mean = 170.25 ms, std = 5.77 ms
difference (B - A) = -17.30 ms
standard error of the difference = 2.05 ms
95 percent CI for the difference = -21.33 to -13.27 ms
Welch t = -8.42, p = 4.61e-10
verdict at alpha 0.05: reject the null, ship B
minimum detectable effect at this noise = 5.75 ms

The verdict is to ship B. The difference is -17.30 ms, the interval is -21.33 to -13.27 ms, and zero is nowhere near it. The p value is 4.61e-10, which is small because the gap is big and the noise is small. The minimum detectable effect is 5.75 ms, so a real change of 17.30 ms is three times bigger than the smallest change this test can see.

Now break it on purpose. Add 15 ms to every value in group B and run it again. The difference shrinks to -2.30 ms and the p value rises to 0.2704. The verdict flips to cannot reject the null. Same test, same noise, smaller gap, different answer. That is what a test is for. It does not tell you B is faster. It tells you whether the data can tell.

Check yourself

Take the lab script and change the sample. Keep only the first 12 latencies and run it. The mean and the median still compute, and the standard error grows, because the error divides by the square root of n. Write down the new standard error and the new interval half width, then compare them with the simulator at 12 requests. Two paths, one number. When two paths agree, the number is worth trusting.

Clean up

Nothing to clean up. The scripts live on your machine, and you can run them again on your own data. Change the latency list to your own week, change the A/B groups to your own change, and read the verdict. The scripts do not call any service and they write no files.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Maths for data work is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many formulas you can name. They are about whether the number you quote survives one hard question. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · One centre, one spread, by hand

At the basic level, you compute the mean, the median, and the standard deviation on one list, by hand or in Python, and you write all three down. You read the p50 and the p95 beside them. The failure that ends this level is the second list. One computed list feels like statistics. It is not. It is a start.

Types:

  • Centre: mean and median.
  • Spread: standard deviation and interquartile range.
  • Percentiles: p50, p95, p99.
  • Distributions: roughly normal versus long-tailed, which latency usually is.

Uses: Describing any metric honestly: latency, cost per request, queue depth.

Scale and failure: Long-tailed data makes the mean misleading: a few slow requests pull it far above what most users see. That is why SLOs use percentiles, and why averaging percentiles across servers gives a wrong answer.

Trade-off: The mean is easy to compute and combine and hides tails; percentiles describe users better and cannot be averaged, so you must keep histograms to combine them.

Do this: run step 0 of the lab. Write down the mean, the median, the standard deviation, the p90, and the p95. Five numbers, one page.

Try it: Basic

I compute the mean, the median, and the standard deviation on one list. Why is that only a start?

Now do it: Run step 0 of the lab and write down the mean, the median, the standard deviation, the p90, and the p95. Five numbers, one page.

Intermediate · The test, run before the decision

At the intermediate level, no change ships on a gap alone. You run the A/B test from the lab, you read the interval, and you check whether zero sits inside it. You fit a regression line and you read the residuals, not just the slope. A number that moves without a test behind it gets a question, not a deploy.

Types:

  • Hypothesis tests and p-values.
  • Confidence intervals for a difference between A and B.
  • Linear regression and residuals.
  • Correlation versus causation.

Uses: Deciding whether a change really moved a metric before shipping it to everyone.

Scale and failure: Running many tests or peeking at results daily inflates false positives; set the analysis plan and sample size before the experiment. Large samples make tiny, irrelevant differences statistically significant, so read the interval's size, not just whether it excludes zero.

Trade-off: Waiting for significance avoids false wins and slows decisions; deciding early is faster and ships more noise.

Do this: run step 1 of the lab, then add 15 ms to every B value and run it again. Write down the two p values, 4.61e-10 and 0.2704, and the two verdicts. Same test, different gap, different answer.

Try it: Intermediate

I run the same A/B test twice and get p values of 4.61e-10 and 0.2704. What changed between the runs?

Now do it: Run step 1 of the lab, then add 15 ms to every B value and run it again. Write down the two p values, 4.61e-10 and 0.2704, and the two verdicts.

Advanced · The forecast, quoted with its interval

At the advanced level, every forecast carries its interval and its expiry date. You fit the trend, you smooth with a moving average, and you say when the forecast stops being useful. You update beliefs with Bayes instead of arguing from one example. A slow request after a deploy is a count, not a story, and the count is 0.60.

Types:

  • Trend fitting and moving averages.
  • Seasonality: daily and weekly cycles in traffic.
  • Prediction intervals for forecasts.
  • Bayes' rule: updating a belief with new evidence.

Uses: Capacity forecasting, anomaly detection on seasonal metrics, and reasoning about rare events.

Scale and failure: Forecasts degrade with distance: a trend fitted on six weeks says little about six months. Seasonal data needs seasonal models, or every Monday morning looks like an anomaly.

Trade-off: Simple models are explainable and miss structure; complex forecasting models capture more and fail in ways that are hard to see. Always publish the interval with the forecast.

Do this: take the seven daily p95 values from the lab and forecast day 21 by hand with the 8.82 ms per day slope. Then write the interval beside it using the standard error. If the interval is wider than the decision, say so in writing.

Try it: Advanced

I forecast day 21 from seven daily p95 values with a slope of 8.82 ms per day. What must I write beside the forecast?

Now do it: Take the seven daily p95 values from the lab and forecast day 21 by hand with the 8.82 ms per day slope. Then write the interval beside it using the standard error.

Master · The maths as a gate, not a slide

At the master level, the maths runs itself. The A/B test runs in CI on every experiment change, with the sample size and the minimum detectable effect computed before the traffic splits. A test that cannot see the effect it claims to measure fails the build. The forecast runs on a schedule, and the alarm fires when the real p95 leaves the forecast interval. The team still holds a review every quarter. The review reads the test history. It does not start from zero.

Types:

  • Power analysis: sample size from the effect you need to detect.
  • Minimum detectable effect, written before the experiment.
  • Sequential testing methods for experiments that are checked continuously.
  • Automated forecast checks that alert when reality leaves the interval.

Uses: Experimentation platforms and capacity planning, where statistical decisions are made many times a week.

Scale and failure: When many teams run experiments, shared standards prevent each one from inventing its own statistics. Automated gates stop underpowered experiments before they waste traffic.

Trade-off: Rigour slows experiments and prevents most false wins; speed ships more changes and more mistakes. The cost of a wrong decision sets the right balance.

Do this: write one check for one rule. Assert the A/B test sample is big enough to see a 10 ms effect at the current noise, using the simulator formula. Run it before the next experiment starts. A test that is only a slide will pass anything. A test that is a gate will hold.

Try it: Master

My A/B test runs in CI on every experiment change. What makes a test that cannot see its effect fail the build?

Now do it: Write one check for one rule. Assert the A/B test sample is big enough to see a 10 ms effect at the current noise, using the simulator formula. Run it before the next experiment starts.

Go deeper

The mean is not the message

The mean is the most quoted number in ops work and the least trustworthy on its own. It moves with every outlier, and it hides the tail. The median holds the centre. The p95 holds the tail. The interquartile range holds the middle half, and it barely moves when one request goes slow. On this week the mean is 215.80 ms, the median is 199.00 ms, the p95 is 261.75 ms, and the interquartile range is 41.25 ms. Four numbers, and the story needs all four. A dashboard that shows only the mean is a dashboard that will miss the next bad deploy, because the deploy shows up in the tail first.

Standard error is not standard deviation

People mix these two up, and the mix-up costs decisions. Standard deviation says how far the requests spread. Standard error says how far the mean might be from the truth. The error divides by the square root of n, so four times the sample halves the error. On the lab list the standard deviation is 89.90 ms and the standard error is 20.10 ms, because n is only 20. On the simulator defaults, 120 requests and a 20 ms spread, the standard error is 1.83 ms. Same kind of system, different sample, and the error falls by a factor of eleven. When someone quotes a mean, ask for n. Without n, the mean is a rumor.

Regression is a claim, and the residuals are the evidence

A regression line is easy to fit and easy to believe. The slope of 2.13 ms per CPU point looks solid, and the prediction of 289.33 ms at 100 percent CPU looks precise. Both are claims, and the residuals are the evidence. A residual is the gap between a real point and the line. If the gaps are small and random, the line fits. If the gaps grow with CPU, the line is the wrong shape, and a straight line through curved data is a lie with a slope. If one point sits far from the line, check that point before you trust the slope. Outliers pull a regression line the way they pull a mean, only harder to see, because the line has no median to fall back on. The code below is the check I run: fit the line, print the residuals, and look at them before I quote the slope.

import numpy as np

cpu = np.array([30., 45., 60., 75., 90.])
lat = np.array([150., 170., 190., 230., 280.])
slope, intercept = np.linalg.lstsq(
    np.vstack([cpu, np.ones(len(cpu))]).T, lat, rcond=None)[0]
pred = slope * cpu + intercept
print("slope = %.2f, intercept = %.2f" % (slope, intercept))
print("residuals =", np.round(lat - pred, 2))

Expected output, exactly as it ran on this machine:

slope = 2.13, intercept = 76.00
residuals = [ 10.  -2. -14.  -6.  12.]

What changes at 10x and 100x

At 10x the sample, 1,200 requests instead of 120, the standard error falls by the square root of 10, to about a third of its old value. The interval narrows, the minimum detectable effect shrinks, and changes that were invisible become visible. That is the cheapest win in statistics: more data, same maths. At 10x the outliers, the mean is lost. No sample size fixes a mean that is owned by bad data. The fix is not more requests. It is naming the outliers, fixing the cause, and quoting the median while you do. At 100x, the work changes shape. The A/B test runs on a schedule with the sample size computed first, the forecast runs daily with its interval, and the maths is a gate in CI, not a slide in a review. The formulas do not change. The number of people reading them does.

Where this goes next

Lesson 184, Classic Machine Learning, builds directly on this lesson. The regression line here becomes the training step there: fit a line, read the slope, check the residuals, and measure the error on data the line has never seen. Precision, recall, F1, and ROC curves are hypothesis testing with a different question: not is the gap real, but how often is the model right when it says yes, and how often is it right when it says no. The Bayes update here becomes the classifier there. The forecast here becomes the time series model there. Learn the maths here, and Lesson 184 is the same tools, pointed at a bigger job.

One time it broke

What broke

A team shipped a change because the dashboard mean dropped 20 ms overnight. The change went to everyone. The next morning the p95 was worse, and the support queue filled up. The mean had dropped because one slow endpoint stopped receiving traffic after a routing change, not because the change made anything faster. The mean measured the traffic mix, not the system. Nobody had looked at the median, the p95, or the sample size. The decision was made on one number, and the number was answering a different question.

Timeline

  • Day 1A change ships to a small group. The mean latency drops 20 ms.
  • Day 2The team reads the mean, calls it a win, and ships to everyone.
  • Day 3The p95 climbs. Users report slow requests. The mean still looks fine.
  • Day 3I open the percentiles. The median barely moved. The p95 moved a lot.
  • Day 3I split the traffic by endpoint. One slow endpoint lost its traffic in the routing change.
  • Day 4The change is rolled back. The mean returns to where it was. The p95 follows.
  • Day 5The change is re-tested as a real A/B test, with the interval and the sample size written down first.

What changed after

Three things. First, the dashboard rule. No mean without a median and a p95 beside it, on the same graph, at the same time. Second, the test rule. No change ships on a gap alone. The interval has to clear zero, and the sample size has to be written down before the traffic splits. Third, the mix rule. Any change that moves traffic between endpoints gets its numbers split by endpoint first. A mean across a changing mix is not a measurement. It is a weather report.

If you remember nothing else

Design exercise

Design the experiment for a new connection pool. The team says it cuts latency by 15 ms. The current mean is 180 ms with a standard deviation of 20 ms. The budget is 100,000 requests for the test. The team wants an answer in one week, and the answer has to be a ship or no-ship decision, not a feeling. Design the A/B test: the split, the sample size, the metric, the decision rule, and what you do when the interval crosses zero.

Clarifying questions to ask

  • Is 15 ms the mean, the median, or the p95. Which number has to move for the change to be worth shipping.
  • How much traffic arrives in a week, and how is it split between the groups. Is 100,000 requests the total, or per group.
  • What is the smallest change worth shipping. If the real gain is 5 ms, is that a win or a waste of a deploy.
  • What else moves during the test week. A deploy, a traffic shift, or a slow endpoint will move the mean, and the test has to survive it.

Worked approach

Start with the metric. The primary metric is the mean latency, and the p95 rides beside it as a guardrail. The mean decides the ship. The p95 vetoes it. If the mean drops 15 ms and the p95 climbs, the change helped the middle and hurt the tail, and the tail is the user. Write both down before the traffic splits.

Then the sample size. The standard error of the difference at 50,000 requests per group is the square root of 400 over 50,000 plus 400 over 50,000, which is 0.13 ms. The minimum detectable effect is 2.8 times that, about 0.35 ms. A 15 ms gain is more than forty times bigger than the smallest effect this test can see. The sample is far bigger than the question needs, which means the test will finish early, or the team can split the traffic unevenly and still decide. If the traffic were only 200 requests per group, the standard error would be 2.00 ms and the minimum detectable effect 5.60 ms, and a 15 ms gain would still be visible. The maths says the budget is generous. Spend the spare on watching the p95, not on more requests.

Then the decision rule. Ship if the 95 percent interval for the difference sits entirely below zero and the p95 did not climb. Do not ship if the interval crosses zero, even if the mean moved in the right direction. An interval that crosses zero is the data saying it cannot tell, and shipping on cannot tell is how the incident in this lesson happened. Write the rule down before the test starts. A rule written after the results is not a rule.

Trade-offs to name. Speed versus certainty is the big one. One week gives a fast answer and a narrow interval at this traffic level. One day gives a faster answer and a wider one. Precision versus the tail is the second. The mean is precise and hides the tail. The p95 shows the tail and jumps around. That is why the mean decides and the p95 vetoes, instead of either one deciding alone. Sample size versus risk is the third. More traffic narrows the interval, and it also exposes more users to a change that might be bad. Split 90 to 10 while the change is unproven, and move to 50 to 50 only when the early numbers hold.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 183
The Maths the Data Roles Assume: CLT, Distributions, Differentiation, Econometrics

Lesson 182 gave me the tools. This lesson gives me the ground they stand on. A derivative is a limit, and I work it by hand before I trust it in code.

Go to Lesson 183 →

Or browse the full course from the Lessons index.