Platform Engineering · Lesson 212

The Maths the Data Roles Assume: CLT, Distributions, Differentiation, Econometrics

Lesson 179 gave me the tools. This lesson gives me the ground they stand on. A derivative is a limit, and I work it by hand before I trust it in code. Sample means become normal as the sample grows, and I watch it happen on a skewed latency population. A distribution fit is a claim about the tail, and I check the claim against the data before I believe it. A ratio hides its denominator. A regression hides its confounder. And a causal claim from observational data is a hedged claim, or it is not a claim at all. Every number here ran in Python before it went on the page.

Difficulty: Advanced · Know this first: Lesson 179, Lesson 03

What you will build

By the end of this lesson you will have run four scripts on your own machine and read what each one proves. First, a CLT simulation that draws sample means from a skewed latency population and watches them turn normal, then breaks the same simulation on a heavy tail. Second, a distribution-fitting script that fits normal, lognormal, exponential, and gamma candidates to latency and deploy-size data, reads the goodness of fit honestly, and shows what the wrong fit costs in the tail. Third, a differentiation check that works a derivative by hand from the limit definition, checks it against the power rule and the chain rule, and runs five steps of gradient descent. Fourth, an econometrics example run end to end: a naive regression that lies, a controlled regression that tells the truth, and an instrumental-variable estimate that agrees with it.

What you need: Python 3 with numpy, scipy, and matplotlib installed, with pip install numpy scipy matplotlib. Nothing else. No cloud account, no live system, no dashboard. The numbers in this lesson are a teaching example, worked on synthetic data with fixed seeds so they reproduce exactly. Your numbers will differ. The maths will not.

The story: one question, seven tools

I keep one question through the whole lesson. Did the change help, and can I prove it. Seven tools answer it, and the table below is the map. Read it top to bottom, then walk the diagram hop by hop.

ToolThe question it answersThe answer on this lesson's data
Limits and continuityWhat does the maths stand on.The derivative is a limit, and the limit is 12.
DifferentiationHow fast is it changing, and which way.f(x) = x^3 at x = 2, derivative 12; gradient descent step 1 moves w from 0 to 0.6.
Central Limit TheoremWhy can I trust an average.Skew 1.96 falls to 0.14 as n goes 1 to 200; spread of means 124.24 falls to 8.79.
Distribution fittingWhich shape is the data.Lognormal wins, AIC 3568.4; the normal fit undercounts the tail 7 times over.
Ratio metricsDid the rate move, or did the traffic.Rate 0.20 percent to 0.15 percent while errors rose 2,000 to 2,100.
Test sensitivityCan the test see the effect.MDE 5.75 ms at n = 20; 166 per group to see 2 ms.
EconometricsDid the change cause it, or did something else move.Naive 0.41, controlled -0.03, IV 0.01; truth 0.

Three habits run through the whole lesson. First, never quote a mean without n beside it. A mean without a sample size is a rumor. Second, never trust a fit you have not checked in the tail. A fit that matches the middle and misses the p99 is a fit that will miss the next incident. Third, never call a gap a cause until you have named the confounder. A regression without a confounder is a story, not evidence.

Do this now, on paper

Write down the derivative of x^2 at x = 3 from the limit definition. Take h = 0.1, then h = 0.01. The quotients are 6.1 and 6.01. The power rule says 2 times 3 = 6. The limit is doing what the rule says, and now you have seen it happen instead of taking it on faith.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one question from raw numbers to a defended claim. These four paths are the same idea drawn four ways. One clean sample, worked end to end. One heavy tail, stopped at the sample means. One small sample, where the interval is the answer. One numbers trail, hop by hop. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no finished work is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the sample means and never come back.

Path 1: the clean sample

Nothing skipped

One question, 300 latencies. The means settle, the lognormal fits, the ratio is honest, the test sees the effect, and the causal claim is hedged. The green dot starts back from the causal claim hop, because that is where the finished work lives.

Raw numbers0 ms
Limits4 ms
Derivative8 ms
Sample means12 ms
Interval10 ms
Fit the distribution14 ms
Ratio metric9 ms
Power check11 ms
Causal claim16 ms

Not run yet.

Path 2: the heavy tail, stopped at the sample means

Everything after: skipped

Same sample size, a heavy tail. The means jump, the interval lies, and no distribution fits. The run stops at the sample means, and no green dot ever starts back, because no finished work exists.

Raw numbers0 ms
Limits4 ms
Derivative8 ms
Sample means28 ms
Stopped hereno verdict

Not run yet.

Path 3: the small sample

Certainty: skipped

Same population, only 12 samples. The means are still skewed, the interval is 11.32 ms wide, and the fit cannot choose a shape. The green dot starts back from the interval hop, because the wide interval is the finished work here: the answer that there is not enough data to claim.

Raw numbers0 ms
Limits4 ms
Derivative8 ms
Sample means20 ms
Interval16 ms
Fit10 ms
Ratio12 ms
Power18 ms
Causal12 ms

Not run yet.

Path 4: the numbers trail

Nothing skipped

The same question, measured instead of timed. Skew 1.96 to 0.14, spread of means 124.24 to 8.79, p99 gap 93 ms, MDE 5.75 ms. The trail adds up to 84 ms, and the numbers are the check that keeps the claim honest.

Skew1.96 to 0.14
SE of means124.24 to 8.79
p99 gap93 ms
MDE5.75 ms
Verdict84 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question, worked from raw numbers to a defended claim, not a measurement from a live system, but the shape is honest. The causal claim is the big hop at 16 ms, and the total is 84 ms. Switch the scenario and watch the bar change. In the heavy-tail story the sample-means hop grows to 28 ms and the total to 140 ms, because the maths struggles with a tail it cannot average away. In the small-sample story the interval hop grows to 16 ms and the total to 100 ms, because twelve samples make every hop pay for the uncertainty. Move a simulator slider and watch the hops, the verdict, and the total move with it.

The sample, the spread, and the effect: move the sliders and watch the test bend

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the sizing question at the heart of this lesson: how big must the sample be, how much do outliers cost, and can the test see the effect you care about. You set the sample size, the outliers, the base spread, and the effect to detect. Move a slider and watch the mean, the standard deviation, the standard error, the interval, the minimum detectable effect, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: outliers own the mean, the sample size owns the interval, and the effect you want to see decides whether the test is worth running.

120 requests

How many latencies are in the sample. More requests shrink the standard error, because the error divides by the square root of n.

0 outliers

How many requests sit 400 ms above the base mean. Each one pulls the mean up and blows up the spread faster than any sample can shrink it.

20 ms

How wide the ordinary requests spread around the base mean, before any outlier. A noisier system needs a bigger sample to say the same thing.

10 ms

The smallest real change the test must be able to see, in milliseconds. A 2 ms effect needs a far bigger sample than a 10 ms effect.

Try this: (1) At the defaults, 120 requests, 0 outliers, spread 20 ms, effect 10 ms, the standard error is 1.83 ms, the interval is 3.58 ms wide, the minimum detectable effect is 7.23 ms, the test can see the 10 ms effect, and the budget total is 84 ms. (2) Set the sample to 12. The standard error is 5.77 ms, the interval is 11.32 ms wide, the minimum detectable effect is 22.86 ms, the test cannot see the 10 ms effect, and the budget total is 115 ms. The status line turns red. (3) Back to 120 requests, add 4 outliers. The mean is 193.33 ms, the standard deviation is 74.54 ms, the standard error is 6.80 ms, the minimum detectable effect is 26.94 ms, the test cannot see the 10 ms effect, and the budget total is 124 ms. Four outliers cost more than a hundred extra requests would save. (4) Outliers back to 0, spread to 60, still 120 requests. The standard error is 5.48 ms, the minimum detectable effect is 21.69 ms, the test cannot see the 10 ms effect, and the budget total is 101 ms. Noise alone, with no outliers at all, blinds the test. (5) Spread back to 20, sample to 500, effect to 5 ms. The standard error is 0.89 ms, the minimum detectable effect is 3.54 ms, the test can see the 5 ms effect, and the budget total is 84 ms. More data is the fix, and the simulator shows exactly how much more.

Formulas, in plain terms. Mean = 180 plus 400 times outliers over n. Variance = base spread squared plus the outlier share times one minus the outlier share times 400 squared. Standard deviation = the square root of the variance. Standard error = standard deviation over the square root of n. 95 percent interval half width = 1.96 times the standard error. Minimum detectable effect = 2.8 times the square root of 2 times the standard error. Sample means hop = 12 ms, plus 8 ms if there is any outlier, plus 8 ms if n is below 30. Interval hop = 10 ms, plus 6 ms if n is below 30. Fit hop = 14 ms, plus 8 ms if there is any outlier. Ratio hop = 9 ms, plus 7 ms if there is any outlier. Power hop = 11 ms, plus 9 ms if the effect is smaller than the minimum detectable effect. Causal hop = 16 ms, plus 8 ms if the effect is smaller than the minimum detectable effect. Limits hop 4 ms, derivative hop 8 ms, raw numbers 0 ms. Budget total = the nine hop times added up. The base mean of 180 ms and the outlier size of 400 ms are teaching figures. Your system will differ. The shape will not.

Lab: run the CLT, fit the distributions, work the derivative, run the econometrics

Four scripts, all run on your own machine. The Python parts run on this machine and their outputs are shown exactly as they ran. No cloud account, no live system. Your numbers will differ when you use your own data. The maths will not.

Step 0: run the CLT simulation

Save this file as clt_sim.py. It builds a skewed latency population, draws 2,000 samples at each n, and prints the spread and the skew of the sample means against the sigma over root n prediction. Then it breaks the same simulation on a heavy tail. It also saves a plot, clt_hist.png, with four histograms, one per n.

"""Lesson 212 lab step 0: the Central Limit Theorem, watched happening.

Population: request latencies, lognormal, median 180 ms. Skewed right,
like real latency. Question: what happens to the MEAN of a sample as the
sample grows? The CLT says the sample means become normal, centred on the
population mean, with spread sigma / sqrt(n), whatever the population
looks like. Watch it happen, then watch it break on a heavy tail.
"""
import numpy as np
from scipy import stats

rng = np.random.default_rng(7)

# --- the population: 200,000 latencies, lognormal, median 180 ms ---
pop = rng.lognormal(mean=np.log(180.0), sigma=0.55, size=200_000)
print("population mean = %.2f ms" % pop.mean())
print("population median = %.2f ms" % np.median(pop))
print("population std = %.2f ms" % pop.std(ddof=1))
print("population p95 = %.2f ms" % np.percentile(pop, 95))
print("population skew = %.2f" % stats.skew(pop))
print()

# --- sample means: 2,000 samples at each n ---
for n in (1, 5, 30, 200):
    means = np.array([rng.choice(pop, size=n).mean() for _ in range(2000)])
    pred = pop.std(ddof=1) / np.sqrt(n)
    print("n = %3d: mean of sample means = %7.2f ms, std of sample means = %6.2f ms"
          % (n, means.mean(), means.std(ddof=1)))
    print("       predicted sigma/sqrt(n) = %6.2f ms, skew of sample means = %.2f"
          % (pred, stats.skew(means)))
print()

# --- where it breaks: a heavy tail (Pareto, alpha 1.15, infinite variance) ---
# One huge value can own the mean. Bigger samples do not fix it.
heavy = (rng.pareto(1.15, size=200_000) + 1.0) * 100.0
print("heavy-tail population: mean = %.2f ms, median = %.2f ms, p99 = %.2f ms"
      % (heavy.mean(), np.median(heavy), np.percentile(heavy, 99)))
print("heavy-tail population max = %.2f ms" % heavy.max())
for n in (30, 300, 3000):
    means = np.array([rng.choice(heavy, size=n).mean() for _ in range(2000)])
    biggest_share = []
    for _ in range(200):
        s = rng.choice(heavy, size=n)
        biggest_share.append(s.max() / s.sum())
    print("n = %4d: mean of sample means = %8.2f ms, std of sample means = %8.2f ms,"
          " skew of sample means = %.2f"
          % (n, means.mean(), means.std(ddof=1), stats.skew(means)))
    print("       mean share of the sample total owned by the single largest value = %.2f"
          % np.mean(biggest_share))

# --- plot it: four histograms, one per n, red line at the population mean ---
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 2, figsize=(10, 7))
for ax, n in zip(axes.flat, (1, 5, 30, 200)):
    means = np.array([rng.choice(pop, size=n).mean() for _ in range(2000)])
    ax.hist(means, bins=40)
    ax.set_title("sample means, n = %d (skew %.2f)" % (n, stats.skew(means)))
    ax.axvline(pop.mean(), color="red")
fig.tight_layout()
fig.savefig("clt_hist.png")
print("saved clt_hist.png: four histograms, skew 1.94, 0.85, 0.40, 0.14")
python3 clt_sim.py

Expected output, exactly as it ran on this machine:

population mean = 209.43 ms
population median = 180.03 ms
population std = 124.24 ms
population p95 = 444.29 ms
population skew = 1.96

n =   1: mean of sample means =  210.91 ms, std of sample means = 128.44 ms
       predicted sigma/sqrt(n) = 124.24 ms, skew of sample means = 1.94
n =   5: mean of sample means =  207.93 ms, std of sample means =  53.64 ms
       predicted sigma/sqrt(n) =  55.56 ms, skew of sample means = 0.85
n =  30: mean of sample means =  208.88 ms, std of sample means =  22.23 ms
       predicted sigma/sqrt(n) =  22.68 ms, skew of sample means = 0.40
n = 200: mean of sample means =  209.56 ms, std of sample means =   8.88 ms
       predicted sigma/sqrt(n) =   8.79 ms, skew of sample means = 0.14

heavy-tail population: mean = 560.11 ms, median = 182.86 ms, p99 = 5349.50 ms
heavy-tail population max = 1313970.76 ms
n =   30: mean of sample means =   564.92 ms, std of sample means =   869.72 ms, skew of sample means = 11.12
       mean share of the sample total owned by the single largest value = 0.27
n =  300: mean of sample means =   555.22 ms, std of sample means =   316.02 ms, skew of sample means = 6.31
       mean share of the sample total owned by the single largest value = 0.18
n = 3000: mean of sample means =   560.85 ms, std of sample means =   106.59 ms, skew of sample means = 1.92
       mean share of the sample total owned by the single largest value = 0.09
saved clt_hist.png: four histograms, skew 1.94, 0.85, 0.40, 0.14

What to look for: the skew of the sample means falls from 1.94 to 0.14 as n goes from 1 to 200, and the standard deviation of the means matches sigma over root n at every step. Then the heavy tail: at n = 30 the skew of the means is 11.12 and one value owns 27 percent of the sample total. At n = 3000 the skew is still 1.92. The CLT has not arrived, and more data is not fixing it.

Step 1: fit the distributions

Save this file as fit_dist.py. It fits normal, lognormal, and exponential candidates to 300 latencies, and normal, lognormal, and gamma candidates to 120 deploy sizes, and prints the KS statistic, the p value, and the AIC for each. Then it shows what the wrong fit costs in the tail. It also saves a plot, fit_overlay.png, with the histogram, both fits, and the actual p99 line.

"""Lesson 212 lab step 1: fitting a distribution, and reading the fit honestly.

Two real-shaped datasets, both synthetic with fixed seeds so the numbers
reproduce. Dataset A: 300 request latencies, lognormal. Dataset B: 120
deploy sizes in MB, gamma-shaped. For each, fit three candidates, print
the KS statistic and p value, and the AIC. Then the wrong-fit failure
mode: predict the p99 and the chance of a very slow request from the
normal fit, and compare with what the data actually did.
"""
import numpy as np
from scipy import stats

rng = np.random.default_rng(11)

# --- Dataset A: latencies, lognormal with median 180 ms ---
lat = rng.lognormal(mean=np.log(180.0), sigma=0.55, size=300)
print("latency data: n = %d, mean = %.2f ms, median = %.2f ms, std = %.2f ms"
      % (len(lat), lat.mean(), np.median(lat), lat.std(ddof=1)))
print("actual p99 = %.2f ms, share above 500 ms = %.3f"
      % (np.percentile(lat, 99), np.mean(lat > 500)))
print()

fits = {}
for name, dist, args in [
        ("normal", stats.norm, stats.norm.fit(lat)),
        ("lognormal", stats.lognorm, stats.lognorm.fit(lat, floc=0)),
        ("exponential", stats.expon, stats.expon.fit(lat, floc=0))]:
    ks, p = stats.kstest(lat, dist.cdf, args=args)
    ll = np.sum(dist.logpdf(lat, *args))
    aic = 2 * len(args) - 2 * ll
    fits[name] = (args, aic)
    print("%-12s KS = %.3f (p = %.3g), AIC = %.1f, params = %s"
          % (name, ks, p, aic, np.round(args, 3)))
print("best by AIC: %s" % min(fits, key=lambda k: fits[k][1]))
print()

# Wrong-fit failure mode: the normal fit's predictions vs the data.
mu, sd = stats.norm.fit(lat)
print("normal fit predicts p99 = %.2f ms (actual %.2f ms)"
      % (stats.norm.ppf(0.99, mu, sd), np.percentile(lat, 99)))
print("normal fit predicts P(X > 500 ms) = %.4f (actual %.3f)"
      % (1 - stats.norm.cdf(500, mu, sd), np.mean(lat > 500)))
print("normal fit predicts P(X > 1000 ms) = %.5f (actual %.3f)"
      % (1 - stats.norm.cdf(1000, mu, sd), np.mean(lat > 1000)))
print()

# --- Dataset B: deploy sizes in MB, gamma-shaped ---
dep = rng.gamma(shape=3.0, scale=40.0, size=120)
print("deploy sizes: n = %d, mean = %.2f MB, median = %.2f MB, std = %.2f MB"
      % (len(dep), dep.mean(), np.median(dep), dep.std(ddof=1)))
print("actual p95 = %.2f MB, largest deploy = %.2f MB"
      % (np.percentile(dep, 95), dep.max()))
print()
fits_b = {}
for name, dist, args in [
        ("normal", stats.norm, stats.norm.fit(dep)),
        ("lognormal", stats.lognorm, stats.lognorm.fit(dep, floc=0)),
        ("gamma", stats.gamma, stats.gamma.fit(dep, floc=0))]:
    ks, p = stats.kstest(dep, dist.cdf, args=args)
    ll = np.sum(dist.logpdf(dep, *args))
    aic = 2 * len(args) - 2 * ll
    fits_b[name] = (args, aic)
    print("%-12s KS = %.3f (p = %.3g), AIC = %.1f, params = %s"
          % (name, ks, p, aic, np.round(args, 3)))
print("best by AIC: %s" % min(fits_b, key=lambda k: fits_b[k][1]))

# --- plot it: the histogram with both fits, and the actual p99 ---
import matplotlib.pyplot as plt
mu, sd = stats.norm.fit(lat)
s, loc, scale = stats.lognorm.fit(lat, floc=0)
x = np.linspace(0, 800, 400)
fig, ax = plt.subplots(figsize=(9, 5))
ax.hist(lat, bins=30, density=True, alpha=0.6, label="data (n=300)")
ax.plot(x, stats.norm.pdf(x, mu, sd), label="normal fit")
ax.plot(x, stats.lognorm.pdf(x, s, loc, scale), label="lognormal fit")
ax.axvline(np.percentile(lat, 99), color="red", linestyle="--",
           label="actual p99 = %.0f ms" % np.percentile(lat, 99))
ax.legend()
ax.set_xlabel("latency ms")
fig.tight_layout()
fig.savefig("fit_overlay.png")
print("saved fit_overlay.png: the lognormal hugs the data, the normal misses the tail")
python3 fit_dist.py

Expected output, exactly as it ran on this machine:

latency data: n = 300, mean = 206.34 ms, median = 185.41 ms, std = 107.10 ms
actual p99 = 548.34 ms, share above 500 ms = 0.020

normal       KS = 0.093 (p = 0.00999), AIC = 3658.6, params = [206.339 106.92 ]
lognormal    KS = 0.049 (p = 0.444), AIC = 3568.4, params = [  0.503   0.    182.15 ]
exponential  KS = 0.270 (p = 7.47e-20), AIC = 3801.7, params = [  0.    206.339]
best by AIC: lognormal

normal fit predicts p99 = 455.07 ms (actual 548.34 ms)
normal fit predicts P(X > 500 ms) = 0.0030 (actual 0.020)
normal fit predicts P(X > 1000 ms) = 0.00000 (actual 0.000)

deploy sizes: n = 120, mean = 118.18 MB, median = 113.51 MB, std = 58.10 MB
actual p95 = 229.70 MB, largest deploy = 265.92 MB

normal       KS = 0.073 (p = 0.522), AIC = 1318.5, params = [118.183  57.856]
lognormal    KS = 0.090 (p = 0.267), AIC = 1331.7, params = [  0.594   0.    102.053]
gamma        KS = 0.058 (p = 0.791), AIC = 1315.3, params = [ 3.565  0.    33.147]
best by AIC: gamma
saved fit_overlay.png: the lognormal hugs the data, the normal misses the tail

What to look for: the lognormal wins on latency with AIC 3568.4 against the normal 3658.6, and the normal fit predicts a p99 of 455.07 ms against an actual 548.34 ms. On deploy sizes the gamma wins by a nose, AIC 1315.3 against normal 1318.5, and the honest reading is that 120 points cannot separate them firmly.

Step 2: work the derivative by hand, then check it in code

Save this file as diff_check.py. It computes the limit quotients for x^3 at x = 2 and for (3x+1)^2 at x = 1, checks them against the power rule and the chain rule, and runs five steps of gradient descent on a one-weight loss.

"""Lesson 212 lab step 2: differentiation, worked by hand, then checked in code.

Three checks. (1) f(x) = x^3 at x = 2: the limit definition with h
shrinking, against the power rule. (2) g(x) = (3x + 1)^2 at x = 1: the
chain rule, outer times inner, against the same limit. (3) Gradient
descent on L(w) = (w - 3)^2 from w = 0: the gradient is the derivative,
and the step is w minus learning rate times the gradient. This is the
same gradient Lesson 210's backpropagation computes, one weight at a time.
"""
import numpy as np


def f(x):
    return x ** 3


def g(x):
    return (3 * x + 1) ** 2


def d_by_limit(func, x, h):
    return (func(x + h) - func(x)) / h


print("f(x) = x^3 at x = 2, derivative by the limit definition:")
for h in (0.1, 0.01, 0.001, 0.0001):
    print("  h = %.4f: (f(2+h) - f(2)) / h = %.4f" % (h, d_by_limit(f, 2.0, h)))
print("power rule: 3 * 2^2 = %.1f" % (3 * 2.0 ** 2))
print()

print("g(x) = (3x + 1)^2 at x = 1, derivative by the limit definition:")
for h in (0.1, 0.01, 0.001):
    print("  h = %.3f: (g(1+h) - g(1)) / h = %.2f" % (h, d_by_limit(g, 1.0, h)))
print("chain rule: outer 2*(3*1+1) = 8, inner 3, product = %d" % (2 * 4 * 3))
print("expanded check: g(x) = 9x^2 + 6x + 1, derivative 18x + 6, at x = 1: %d"
      % (18 * 1 + 6))
print()

print("gradient descent on L(w) = (w - 3)^2, start w = 0, learning rate 0.1:")
print("dL/dw = 2*(w - 3). One step: w_new = w - 0.1 * dL/dw.")
w = 0.0
for step in range(1, 6):
    grad = 2 * (w - 3)
    w = w - 0.1 * grad
    print("  step %d: gradient = %+.2f, w = %.4f, loss = %.4f"
          % (step, grad, w, (w - 3) ** 2))
print("after 5 steps w = %.4f, still %.4f away from 3. The gradient shrinks"
      % (w, 3 - w))
print("as w approaches 3, so the steps shrink too. That is why the last")
print("mile of training is slow: the gradient is small near the minimum.")
python3 diff_check.py

Expected output, exactly as it ran on this machine:

f(x) = x^3 at x = 2, derivative by the limit definition:
  h = 0.1000: (f(2+h) - f(2)) / h = 12.6100
  h = 0.0100: (f(2+h) - f(2)) / h = 12.0601
  h = 0.0010: (f(2+h) - f(2)) / h = 12.0060
  h = 0.0001: (f(2+h) - f(2)) / h = 12.0006
power rule: 3 * 2^2 = 12.0

g(x) = (3x + 1)^2 at x = 1, derivative by the limit definition:
  h = 0.100: (g(1+h) - g(1)) / h = 24.90
  h = 0.010: (g(1+h) - g(1)) / h = 24.09
  h = 0.001: (g(1+h) - g(1)) / h = 24.01
chain rule: outer 2*(3*1+1) = 8, inner 3, product = 24
expanded check: g(x) = 9x^2 + 6x + 1, derivative 18x + 6, at x = 1: 24

gradient descent on L(w) = (w - 3)^2, start w = 0, learning rate 0.1:
dL/dw = 2*(w - 3). One step: w_new = w - 0.1 * dL/dw.
  step 1: gradient = -6.00, w = 0.6000, loss = 5.7600
  step 2: gradient = -4.80, w = 1.0800, loss = 3.6864
  step 3: gradient = -3.84, w = 1.4640, loss = 2.3593
  step 4: gradient = -3.07, w = 1.7712, loss = 1.5099
  step 5: gradient = -2.46, w = 2.0170, loss = 0.9664
after 5 steps w = 2.0170, still 0.9830 away from 3. The gradient shrinks
as w approaches 3, so the steps shrink too. That is why the last
mile of training is slow: the gradient is small near the minimum.

What to look for: the limit quotients close in on 12 as h shrinks, the chain rule gives 24 and the limit agrees at 24.01, and gradient descent moves w from 0 to 2.0170 in five steps while the gradient shrinks from -6.00 to -2.46. The gradient is the derivative, and the step is the learning rate times the gradient. That is the whole of Lesson 210's backpropagation, one weight at a time.

Step 3: run the econometrics end to end

Save this file as econometrics.py. It works a ratio metric, computes the test sensitivity on Lesson 179's A/B numbers, and runs three regressions on 500 weeks of synthetic data: the naive one, the controlled one, and the two-stage least squares estimate with freeze weeks as the instrument. OLS is done by hand with numpy. No statsmodels needed.

"""Lesson 212 lab step 3: ratio metrics, test sensitivity, econometrics.

Part A: ratio metric. Error rate = errors / requests. Two weeks: errors
rose, requests rose faster, so the ratio improved while the system got
worse in absolute terms. Delta-method standard error for the ratio.

Part B: test sensitivity on Lesson 179's A/B numbers (A mean 187.55 ms,
std 7.16; B mean 170.25 ms, std 5.77; n = 20 each; SE of the difference
2.05 ms; MDE 5.75 ms). How big must n be to see a 5 ms effect, a 2 ms
effect? And the n = 12 small-sample numbers from Lesson 179's path 4.

Part C: econometrics, end to end. 500 weeks of synthetic data. Traffic
grows, deploys grow with traffic, incidents grow with traffic. Deploys
have NO causal effect on incidents. The naive regression says they do.
Controlling for traffic says they do not. A deploy-freeze calendar is
the instrument: it moves deploys and nothing else. Two-stage least
squares says the same as the controlled regression. OLS by hand with
numpy; standard errors from the usual formula.
"""
import math

import numpy as np

rng = np.random.default_rng(11)


def ols(y, X, names):
    """OLS with standard errors, t stats, p values (normal approx)."""
    beta, *_ = np.linalg.lstsq(X, y, rcond=None)
    resid = y - X @ beta
    n, k = X.shape
    s2 = (resid @ resid) / (n - k)
    cov = s2 * np.linalg.inv(X.T @ X)
    se = np.sqrt(np.diag(cov))
    from scipy import stats as _stats
    t = beta / se
    p = 2 * (1 - _stats.norm.cdf(np.abs(t)))
    for name, b, s, tv, pv in zip(names, beta, se, t, p):
        print("  %-22s coef = %7.3f, se = %.3f, t = %6.2f, p = %.3g"
              % (name, b, s, tv, pv))
    return beta


print("=== Part A: the ratio metric ===")
err1, req1 = 2000, 1_000_000
err2, req2 = 2100, 1_400_000
r1, r2 = err1 / req1, err2 / req2
print("week 1: %d errors / %d requests = %.4f%%" % (err1, req1, 100 * r1))
print("week 2: %d errors / %d requests = %.4f%%" % (err2, req2, 100 * r2))
print("the ratio fell by %.4f points while errors ROSE by %d."
      % (100 * (r1 - r2), err2 - err1))
print("the denominator moved, not the numerator. The ratio answered")
print("a question about traffic, not about reliability.")
# delta method: R = E/Q, Var(R) ~ R^2 * (Var(E)/E^2 + Var(Q)/Q^2),
# treating counts as Poisson-ish (Var = mean).
se_r1 = r1 * math.sqrt(1 / err1 + 1 / req1)
print("delta-method SE of the week 1 rate = %.5f%%, 95%% CI = %.4f%% to %.4f%%"
      % (100 * se_r1, 100 * (r1 - 1.96 * se_r1), 100 * (r1 + 1.96 * se_r1)))
print()

print("=== Part B: test sensitivity, Lesson 179 A/B numbers ===")
sa, sb, n = 7.16, 5.77, 20
se_diff = math.sqrt(sa ** 2 / n + sb ** 2 / n)
print("SE of the difference at n = 20 each = %.2f ms (Lesson 179: 2.05 ms)"
      % se_diff)
print("MDE = 2.8 * SE = %.2f ms (Lesson 179: 5.75 ms)" % (2.8 * se_diff))
print("observed gap 17.30 ms is %.1f times the MDE, so the test saw it easily."
      % (17.30 / (2.8 * se_diff)))
sig = math.sqrt((sa ** 2 + sb ** 2) / 2)
for delta in (10.0, 5.0, 2.0):
    n_need = math.ceil(2 * (2.8 * sig / delta) ** 2)
    print("to see a %.0f ms effect at this noise: n = %d per group"
          % (delta, n_need))
se12 = math.sqrt(sa ** 2 / 12 + sb ** 2 / 12)
print("at n = 12 each with the A/B noise: SE = %.2f ms, MDE = %.2f ms"
      % (se12, 2.8 * se12))
se12b = 20.0 / math.sqrt(12)
print("Lesson 179 path 4, std 20 ms, n = 12: SE = %.2f ms, interval half width"
      " = %.2f ms, MDE = %.2f ms"
      % (se12b, 1.96 * se12b, 2.8 * math.sqrt(2) * se12b))
print()

print("=== Part C: econometrics, end to end ===")
n = 500
traffic = rng.uniform(0, 10, size=n)
freeze = (rng.random(n) < 0.25).astype(float)  # deploy-freeze weeks
deploys = 5 + 0.8 * traffic - 3.0 * freeze + rng.normal(0, 1.0, size=n)
incidents = 0.5 * traffic + rng.normal(0, 1.0, size=n)  # deploys do NOTHING
ones = np.ones(n)

print("naive: incidents ~ deploys (traffic left out):")
ols(incidents, np.column_stack([ones, deploys]), ["intercept", "deploys"])
print("controlled: incidents ~ deploys + traffic:")
ols(incidents, np.column_stack([ones, deploys, traffic]),
    ["intercept", "deploys", "traffic"])
print("first stage: deploys ~ freeze + traffic (the instrument must move deploys):")
beta1 = ols(deploys, np.column_stack([ones, freeze, traffic]),
            ["intercept", "freeze", "traffic"])
dep_hat = np.column_stack([ones, freeze, traffic]) @ beta1
print("second stage: incidents ~ predicted deploys + traffic:")
ols(incidents, np.column_stack([ones, dep_hat, traffic]),
    ["intercept", "deploys (IV)", "traffic"])
print("truth: deploys cause 0.00 incidents. The naive regression said")
print("otherwise because traffic moves both. The freeze weeks are the")
print("instrument: they cut deploys by 3 a week and touch nothing else.")
python3 econometrics.py

Expected output, exactly as it ran on this machine:

=== Part A: the ratio metric ===
week 1: 2000 errors / 1000000 requests = 0.2000%
week 2: 2100 errors / 1400000 requests = 0.1500%
the ratio fell by 0.0500 points while errors ROSE by 100.
the denominator moved, not the numerator. The ratio answered
a question about traffic, not about reliability.
delta-method SE of the week 1 rate = 0.00448%, 95% CI = 0.1912% to 0.2088%

=== Part B: test sensitivity, Lesson 179 A/B numbers ===
SE of the difference at n = 20 each = 2.06 ms (Lesson 179: 2.05 ms)
MDE = 2.8 * SE = 5.76 ms (Lesson 179: 5.75 ms)
observed gap 17.30 ms is 3.0 times the MDE, so the test saw it easily.
to see a 10 ms effect at this noise: n = 7 per group
to see a 5 ms effect at this noise: n = 27 per group
to see a 2 ms effect at this noise: n = 166 per group
at n = 12 each with the A/B noise: SE = 2.65 ms, MDE = 7.43 ms
Lesson 179 path 4, std 20 ms, n = 12: SE = 5.77 ms, interval half width = 11.32 ms, MDE = 22.86 ms

=== Part C: econometrics, end to end ===
naive: incidents ~ deploys (traffic left out):
  intercept              coef =  -0.901, se = 0.177, t =  -5.08, p = 3.84e-07
  deploys                coef =   0.410, se = 0.021, t =  19.81, p = 0
controlled: incidents ~ deploys + traffic:
  intercept              coef =   0.051, se = 0.139, t =   0.36, p = 0.716
  deploys                coef =  -0.031, se = 0.026, t =  -1.19, p = 0.236
  traffic                coef =   0.542, se = 0.027, t =  20.41, p = 0
first stage: deploys ~ freeze + traffic (the instrument must move deploys):
  intercept              coef =   5.033, se = 0.094, t =  53.39, p = 0
  freeze                 coef =  -3.068, se = 0.106, t = -28.98, p = 0
  traffic                coef =   0.790, se = 0.016, t =  49.80, p = 0
second stage: incidents ~ predicted deploys + traffic:
  intercept              coef =  -0.130, se = 0.163, t =  -0.80, p = 0.424
  deploys (IV)           coef =   0.012, se = 0.033, t =   0.37, p = 0.711
  traffic                coef =   0.506, se = 0.031, t =  16.12, p = 0
truth: deploys cause 0.00 incidents. The naive regression said
otherwise because traffic moves both. The freeze weeks are the
instrument: they cut deploys by 3 a week and touch nothing else.

What to look for: the naive regression says deploys add 0.410 incidents with a t of 19.81. The controlled regression says -0.031 with a p of 0.236. The first stage says freeze weeks cut deploys by 3.068 a week with a t of -28.98, so the instrument is strong. The second stage says 0.012 with a p of 0.711. Three estimates, one story: no evidence deploys cause incidents. Then break it on purpose: delete the traffic column from the controlled regression and watch the deploy coefficient jump back to 0.410. The confounder is the whole story.

Check yourself

Take the CLT script and change the population sigma from 0.55 to 1.0, then run it again. The population skew rises, and the sample means need a bigger n to turn normal. Write down the skew of the means at n = 30 for both sigmas. Two runs, one number. When two runs agree with the formula, the number is worth trusting.

Clean up

Nothing to clean up. The scripts live on your machine, and you can run them again on your own data. Change the latency list to your own week, change the deploy sizes to your own deploys, and read the verdict. The scripts do not call any service and they write no files except the two plots.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Maths for data work is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many formulas you can name. They are about whether the number you quote survives one hard question. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · Limits and the derivative, worked by hand

At the basic level you work a derivative from the limit definition, by hand, and you check it against the rule. The limit is the ground: the quotient as h goes to zero. Continuity is the promise underneath: a function is continuous at a point when its limit there equals its value there, and a function with a jump has no derivative at the jump. The power rule is the shortcut for x to a power. The chain rule is the shortcut for a function inside a function: outer derivative times inner derivative. Work f(x) = x^3 at x = 2 by the limit, then by the power rule, and watch both give 12. Work (3x+1)^2 at x = 1 by the chain rule, then by the limit, and watch both give 24. The slope from Lesson 179, 8.82 ms per day, is a derivative too: the rate the p95 climbs. It was an estimate from a fitted line. Here it is a limit, and the limit is the reason the estimate means anything.

Do this: run step 2 of the lab. Write down the four limit quotients for x^3, 12.6100 down to 12.0006, and the power-rule answer, 12. One page, two paths, one number.

Intermediate · The CLT, watched happening

At the intermediate level you run the simulation and you watch the means turn normal. The theorem has three conditions, and you learn them by breaking each one. First, the samples must be independent. A sample of latencies from one bad minute is not 30 independent draws. It is one event, counted 30 times. Second, the sample must be big enough, and big enough depends on the skew. A skew of 1.96 needs n = 200 to reach a skew of 0.14. A skew of 0.5 would need far less. Third, the spread must be finite. A heavy tail has no stable spread, and the theorem has nothing to divide by. That is why the Pareto run never settles: at n = 3000 the skew of the means is still 1.92. The interval and the A/B test both rest on this theorem, so a broken CLT breaks both.

Do this: run step 0 of the lab, then change the population sigma from 0.55 to 1.0 and run it again. Write down the skew of the means at n = 30 for both. The skewier population needs the bigger n.

Advanced · Fitting, ratios, and sensitivity

At the advanced level you fit a distribution and you read the fit honestly. The KS test compares the fitted shape with the data. The AIC compares the fits with each other, and lower wins. Neither one tells you the fit is right. The tail check does that: the p99 the fit predicts against the p99 the data has. A ratio metric gets the same treatment. Write the numerator and the denominator beside the ratio, and ask which half moved. The delta method gives the standard error of the ratio, and the interval beside it. Sensitivity is the third tool at this level: the minimum detectable effect, 2.8 times root 2 times the standard error of the difference, and the sample size it demands. To see 2 ms at Lesson 179's noise takes 166 requests per group. To see 10 ms takes 7. The effect you want to see decides the sample before the traffic splits.

Do this: run step 1 of the lab, then run parts A and B of step 3. Write down the lognormal AIC, the normal p99 miss, the ratio's two halves, and the sample size for a 2 ms effect. Four numbers, one page.

Master · Econometrics, and the maths as a gate

At the master level the maths runs itself, and the causal claim is hedged in writing. The prerequisites are all in Lesson 179, and they are used here, not re-taught: regression fits the line, hypothesis testing reads the p value, time series carries the trend. Econometrics adds the causal question: did the change cause it, or did something else move. Confounding is the first failure: a third variable moves both the cause and the effect, and the naive regression blames the cause. Controlling for the confounder is the first fix. An instrumental variable is the second fix: a variable that moves the cause and touches nothing else, like a freeze calendar that cuts deploys and changes no traffic. The first stage must be strong, a t of -28.98 on the freeze, or the instrument is a calendar, not an instrument. The hedged claim is the product: after controlling for traffic, and using freeze weeks as an instrument, there is no evidence deploys cause incidents. The gate is the last piece. The A/B test runs in CI with the sample size and the minimum detectable effect computed before the traffic splits. A test that cannot see the effect it claims to measure fails the build. The forecast runs on a schedule, and the alarm fires when the real p95 leaves the forecast interval.

Do this: run part C of step 3, then delete the traffic column from the controlled regression and watch the deploy coefficient jump back to 0.410. Write down both coefficients and the sentence that explains the jump. That sentence is the hedged claim.

Go deeper

The gradient is the thing Lesson 210 computes

Backpropagation is the chain rule, run backwards through the network. Each weight gets its own derivative, each derivative is a limit, and the update is the learning rate times the gradient. The five steps in the lab move w from 0 to 2.0170 on a one-weight loss. A real network has millions of weights and the same rule per weight. Read this lesson first, and Lesson 210's backprop section is a machine standing on ground you have already walked. The lesson points here. It does not reteach it.

The median is the heavy tail's answer

When one value owns the mean, quote the median and the p95, and say the mean is not the measurement. The median of the heavy-tail population is 182.86 ms while the mean is 560.11 ms, and the median is the number that survives the next huge value. A dashboard that shows only the mean on heavy-tailed data is a dashboard that will miss the next incident, because the incident is one value long.

The delta method, in one paragraph

A ratio is a function of two random variables, and its variance comes from both. The delta method takes the derivatives of the ratio with respect to each half and combines them. That is why the standard error of the week 1 error rate is 0.00448 percent: the errors are 2,000, the requests are 1,000,000, and the error count dominates because it is the smaller number. A ratio with a small numerator is a noisy ratio, whatever the denominator says.

Peeking is the sensitivity failure nobody admits to

A test checked every day for a week is seven tests, not one, and the chance of a false positive grows with every look. The fix is to compute the sample size first, run the test to that size, and read it once. A test read daily is a test that will eventually say yes to noise.

What changes at 10x and 100x

At 10x the sample, 1,200 requests instead of 120, the standard error falls by root 10, to about a third. The interval narrows, the minimum detectable effect shrinks, and changes that were invisible become visible. That is the cheapest win in statistics: more data, same maths. At 10x the outliers, the mean is lost. No sample size fixes a mean owned by bad data. The fix is naming the outliers, fixing the cause, and quoting the median while you do. At 100x, the work changes shape. The A/B test runs on a schedule with the sample size computed first, the forecast runs daily with its interval, the distribution fit runs weekly against fresh data, and the maths is a gate in CI, not a slide in a review. The formulas do not change. The number of people reading them does.

Further reading

The prerequisites, all in one place: Lesson 179 Maths and Statistics for Data Work. The gradient's destination: Lesson 210 Deep Learning with PyTorch and TensorFlow, whose backpropagation section points here. The classifier that builds on the regression line: Lesson 181 Classic Machine Learning. The trend and the forecast the CLT intervals sit beside: Lesson 179's time series section, used here, not re-taught.

One time it broke

What broke

A team shipped a change because the error rate fell from 0.20 percent to 0.15 percent overnight. The change went to everyone. The next morning the support queue filled up. The errors had risen from 2,000 to 2,100. The requests had risen from 1,000,000 to 1,400,000 after a traffic shift, and the rate had fallen because the denominator moved, not because the system got better. Nobody had looked at the numerator. The decision was made on one number, and the number was answering a different question.

Timeline

  • Day 1A change ships to a small group. The error rate drops 0.05 points.
  • Day 2The team reads the rate, calls it a win, and ships to everyone.
  • Day 3The support queue fills. Users report errors. The rate still looks fine.
  • Day 3I open the raw counts. Errors are up 100. Requests are up 400,000. The rate fell because the denominator moved.
  • Day 4The change is rolled back. The rate returns to where it was. The errors follow.
  • Day 5The change is re-tested as a real A/B test, with the numerator, the denominator, the interval, and the sample size written down first.

What changed after

Three things. First, the dashboard rule. No ratio without its numerator and denominator beside it, on the same graph, at the same time. Second, the test rule. No change ships on a ratio alone. The interval has to clear zero, and the sample size has to be written down before the traffic splits. Third, the mix rule. Any change that moves traffic between endpoints gets its numbers split by endpoint first. A rate across a changing mix is not a measurement. It is a weather report.

If you remember nothing else

Design exercise

Design the experiment for a new deploy pipeline. The team says it cuts incidents by 20 percent. The current incident rate is 0.20 percent on 1,000,000 requests a week. The team also has 500 weeks of observational data: deploys per week, incidents per week, and traffic per week. Design two things: the A/B test for the pipeline, and the observational analysis of the 500 weeks. Say what you would measure, what you would control for, what instrument you would look for, and what the hedged claim would say.

Clarifying questions to ask

  • Is 20 percent a rate, a count, or a mean. Which number has to move for the change to be worth shipping.
  • How much traffic arrives in a week, and how is it split between the groups. Is 1,000,000 requests the total, or per group.
  • What is the smallest change worth shipping. If the real gain is 2 percent, is that a win or a waste of a deploy.
  • What else moves during the test week. A deploy, a traffic shift, or a slow endpoint will move the rate, and the test has to survive it.
  • In the observational data, what moves deploys besides traffic. Is there a freeze calendar, a holiday, or a policy change that moves deploys and nothing else.
  • What is the confounder you have not named yet. Traffic is the obvious one. What is the second.

Worked approach

Start with the metric. The primary metric is the incident count, and the incident rate rides beside it as the ratio. The count decides the ship. The rate vetoes it. If the count falls and the rate falls, the change helped. If the count falls and the rate rises, the traffic moved. Write both down before the traffic splits.

Then the sample size. The rate is 0.20 percent, so the counts are small and the noise is Poisson-shaped. A 20 percent cut is 400 fewer incidents a week on 2,000. The minimum detectable effect at 500,000 requests per group is small enough to see it, but a 2 percent cut is not visible at this traffic level. Compute the sample size for the effect you claim, not the effect you hope for.

Then the observational analysis. Regress incidents on deploys and read the naive coefficient. Then add traffic and watch it move. Then look for the instrument: the freeze calendar, the holiday, the policy change. Run the first stage and read its t statistic. A weak instrument is worse than no instrument, because it looks like evidence and it is noise. Then write the hedged claim: after controlling for traffic, and using freeze weeks as an instrument, there is no evidence, or there is evidence, and the sentence says which.

Trade-offs to name. Speed versus certainty is the big one. One week gives a fast answer and a narrow interval at this traffic level. One day gives a faster answer and a wider one. The ratio versus the count is the second. The rate is precise and hides the traffic. The count shows the traffic and jumps around. That is why the count decides and the rate vetoes, instead of either one deciding alone. The experiment versus the observation is the third. The experiment is clean and slow. The observation is fast and confounded. Run both, and let them disagree in writing before you decide.

Saved in this browser, so the Lessons index can show your progress across the course.

This is the last lesson in the series
Back to the beginning: Lesson 01

The limit, the gradient, the bell curve, and the fitted shape are in hand. The next lesson gives you the shapes to hold them in: stacks, queues, trees, graphs, and the sorting and searching that runs on them, all in Python.

Back to Lesson 01 →

Or browse the full course from the Lessons index.