System Design · Lesson 87

Model Compression

Here is the plain version first. A model is a big file of numbers. A 7 billion parameter model in full precision is about 14 gigabytes of numbers, and it needs a big expensive GPU to answer fast. Model compression is the work of making that file smaller and faster without making the answers much worse. It exists because the big model is often more model than the job needs. I run small models on my own machine with Ollama for my k8s-ai-sre project, and I feel the same problem at my desk that a serving team feels in production. The big model answers well and costs too much to run for every user. The compressed model answers almost as well, fits on a cheaper GPU, and answers faster. This lesson takes one 7 billion parameter model through the whole shrink: measure it, quantize it, prune it, distill it, retrain a little around the damage, test it, and ship it. The numbers in this lesson are a teaching example. The trade-offs are the real thing.

Difficulty: Advanced · Know this first: Lesson 64, Lesson 57

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every step charges rent in minutes, not milliseconds. The bar below is a worked example for one compression job on a 7 billion parameter model, not a measurement from a live system, and your numbers will differ. The shape is honest. The shrinking steps are fast. The repair steps are slow. Quantizing takes 45 minutes. Retraining around the damage takes two hours. Step through the lesson or switch the scenario and watch the bar change. The squeezed run spends its time in the eval that catches the damage and the ship step that rolls it back. The quantize-only run skips the slow steps entirely and finishes in about two hours.

Change the shrink and watch the trade move

The budget above is three fixed stories. This box is yours. Pick a precision, prune harder, turn distillation off, add retraining epochs, and watch the size, the speed, and the quality move together. Then ask the question I ask before every compression job: what am I paying for, and what is it buying me?

The bytes per number, so the model size, the speed, and the quality loss. FP16 is 2 bytes per number. INT8 is 1 byte. INT4 is 0.5 bytes. Size in GB is 7 times bytes per number times 1 minus the prune share.

More pruning means a smaller model and a bigger quality loss. Quality loses 0.2 points for every percent pruned. Pruning 20 percent costs 4 quality points. Pruning 40 percent costs 8.

The quality only. Distillation recovers 2 quality points by teaching the student the teacher's style. It costs about 90 minutes of job time and does not change the size.

The quality only. Each epoch of quantization-aware retraining recovers 1 quality point, up to 3. Two epochs recover 2 points. Zero epochs recovers nothing.

Model size5.6 GB7 x bytes per number x (1 minus prune share)
Speedabout 15 ms / token8 + 1.2 x size in GB
Quality (golden set)80 / 10082 minus quant loss minus prune loss plus distill plus QAT
1,000 requests (60 tokens each)900 s · 15 minGPU decode seconds, 60 x ms per token

Try this: (1) Set precision to INT4, prune to 40 percent, distillation on, QAT epochs to 0. Size is 2.1 GB, speed is about 11 ms per token, quality is 69. Small and fast and below the floor. (2) Now set precision to FP16, prune to 0, distillation off, QAT epochs to 0. Size is 14 GB, speed is about 25 ms per token, quality is 82. That is the big model again, unchanged. (3) Back to the defaults, then turn distillation off. Quality drops from 80 to 78. Two points, and that is exactly what the teacher was worth.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The same topic four times, from the first plain idea to the level where I run it in production and debug it at 2 AM. Each level adds one thing the level below cannot do.

Basic — What it is

Definition: A model is a big file of numbers. A 7 billion parameter model in full precision is about 14 gigabytes of numbers, and it needs a big expensive GPU to answer fast. Model compression is the work of making that file smaller and faster without making the answers much worse. It exists because the big model is often more model than the job needs.

Types:

  • Quantization: smaller numbers, fewer bytes each.
  • Pruning: whole pieces out.
  • Distillation: a big teacher teaches a small student.
  • QAT: retraining a little around the damage.

Uses: Use it when the model is too big for the box you have. I run small models on my own machine with Ollama for my k8s-ai-sre project, and I feel the same problem at my desk that a serving team feels in production. The big model answers well and costs too much to run for every user. The compressed model answers almost as well, fits on a cheaper GPU, and answers faster.

Dimensions: At low scale one model on one box is simple. At 10x the traffic the serving box is the bottleneck, not the compression job. What breaks is a compressed model with no fast kernel for its precision: INT8 with no fast kernel can run slower than FP16 with a good one, so I test the speed on the real box before I promise it. It costs the compression job, the eval runs, and the second GPU you no longer need. You give up a little quality for a lot of cheapness. The curve decides how much, not your gut.

Intermediate — Quantization and the curve

Definition: Quantization stores each number in fewer bytes. FP16 to INT8 halves the model and costs about 2 quality points. INT8 to INT4 halves it again and costs about 5 more. Pruning the first 20 percent costs about 4 points. Pruning the next 20 percent costs about 8 more. The curve bends, and the bend is the whole decision.

Types:

  • GPTQ: rounds each number by asking how much that rounding changes the model's output, and fixes the error in the numbers next to it. It spreads the damage thin and even.
  • AWQ: looks for the few numbers that matter most, the ones that light up hardest on real prompts, and protects those by scaling them up before rounding.
  • Calibration prompts: 500 real ones, long ones included, because the calibration decides what the quantizer protects.
  • The accuracy-vs-size curve: size on one axis, quality on the other, one point per setting.

Uses: Use it before picking a method. I pick AWQ when the model has a small set of loud numbers, which most models do. I pick GPTQ when the numbers are flatter and there is no loud set to protect. Either way, I never pick the method on a benchmark. I pick it on my calibration prompts and my golden set, because the method that wins on a public test can lose on my traffic.

Dimensions: The compressed weights are smaller, so more of them fit in GPU memory and the decode step reads them faster. That is the win. But the KV cache does not shrink with the weights. The notes the model takes while reading are still full size unless you quantize those too, and quantizing the KV cache is a separate choice with its own quality cost. I size the serving box with both numbers: weights plus KV cache for the concurrent requests I actually expect. What breaks is a calibration set with only short prompts: it repairs short prompts and leaves the long ones broken. It costs the calibration run, the eval per setting, and the kernel check. You give up the first half of the shrinking, which is nearly free, for the second half, which is expensive. Pick the point just before the bend.

Advanced — Pruning, distillation, and QAT

Definition: Pruning removes whole pieces so the shape stays clean and the GPU actually runs faster. Distillation teaches the student the teacher's style, not its facts. QAT works because the training sees the damage while it can still fix it. The model trains with the small numbers in place, and every update is rounded the same way the deployed model will be rounded.

Types:

  • Structured pruning: whole pieces out, 20 percent in this lesson, so the shape stays clean.
  • Distillation: teacher uncompressed, student the small model.
  • QAT: two epochs, learning rate 0.0001, low on purpose, because the rounding fights big steps.
  • Split evals: by task and by prompt length, because the overall score hides uneven damage.

Uses: Use it when quantization alone does not fit the box. The config in this lesson targets weights under 8 GB and a golden score of 78 or better, and every step serves that target. AWQ protects the loud numbers before rounding. Structured pruning keeps the shape clean. QAT runs two epochs at a low learning rate so the small numbers learn without chasing their own tail. And the base model stays deployed behind a flag until the small one has survived a week.

Dimensions: At 10x the models the compression pipeline is the bottleneck, so I automate it: one config, one calibration set, one golden set per model, and the same eval gates on every run. What breaks is a high learning rate in QAT: it moves the numbers fast and the rounding throws them back, so the model chases its own tail. I run QAT for two epochs by default, check the golden set after each one, and stop at the peak. It costs the teacher's compute, the QAT epochs, and the evals after each epoch. You give up the base model's quality for a smaller model that fits. If the golden set says the small model fails the hard cases no matter the settings, stop shrinking and keep the bigger model.

Master — Compression as a service

Definition: At 100x, compression is a service with versions. Every compressed model carries its recipe: which precision, which prune, which teacher, which QAT epochs, and which golden set score it got. I can answer the question "which recipe made this model" a year later, because the answer is in the log, not in someone's memory.

Types:

  • The recipe: versioned, logged, reproducible.
  • The eval gates: the same gates on every run.
  • The keep-base rule: one click back until the small model survives a week.
  • The serving check: weights plus KV cache plus kernels, on the real box.

Uses: Use it when compression is a routine, not a project. A smaller model that fits on one cheap GPU beats a bigger model that needs two expensive ones, if the quality holds. That is the common case. But there is a line. If the job needs rare knowledge, long reasoning, or judgment the small model never had, no amount of compression brings it back. Distillation cannot teach what the teacher barely knows. QAT cannot repair knowledge that was never in the student.

Dimensions: At 100x the users mean ten times the notes in GPU memory, so batching and the KV cache matter more than the next compression trick. At 10x the traffic, I compress once, then I spend my time on batching and the KV cache. What breaks is uneven damage the overall score hides: the split eval is the defense. It costs the service, the registry, and the eval harness. You give up hand-tuned compression for a pipeline that is boring and repeatable. Compression is a tool for the jobs where the big model was more than the job needed. It is not a way to make a small model smart.

Go deeper

The lesson above is one model through nine hops. Production is the rest of it. Here is what I actually spend my time on once the first compressed model ships and the second job starts.

GPTQ vs AWQ, in plain terms

Both methods shrink the numbers, and they disagree about which numbers to protect. GPTQ rounds each number by asking how much that rounding changes the model's output on the calibration prompts, and it fixes the error in the numbers next to it as it goes. It spreads the damage thin and even. AWQ looks for the few numbers that matter most, the ones that light up hardest on real prompts, and it protects those by scaling them up before rounding, so the rounding hurts them less. I pick AWQ when the model has a small set of loud numbers, which most models do. I pick GPTQ when the numbers are flatter and there is no loud set to protect. Either way, I never pick the method on a benchmark. I pick it on my calibration prompts and my golden set, because the method that wins on a public test can lose on my traffic.

The accuracy vs size curve

The first half of the shrinking is nearly free. FP16 to INT8 halves the model and costs about 2 quality points. The second half is expensive. INT8 to INT4 halves it again and costs about 5 more points. Pruning the first 20 percent costs about 4 points. Pruning the next 20 percent costs about 8 more. The curve bends, and the bend is the whole decision. I draw the curve for my own model: size on one axis, quality on the other, one point per setting. Then I pick the point just before the bend, where the next gigabyte saved starts costing real quality. A team that skips the curve picks a setting by feel, and feel always picks too small or too big.

Quantization-aware fine-tuning, the repair step

QAT works because the training sees the damage while it can still fix it. The model trains with the small numbers in place, and every update is rounded the same way the deployed model will be rounded. Two things matter. First, the learning rate stays low. A high rate moves the numbers fast and the rounding throws them back, so the model chases its own tail. Second, the training data must include the hard cases, not just the easy ones. If QAT only sees short prompts, it repairs short prompts and leaves the long ones broken. I run QAT for two epochs by default, check the golden set after each one, and stop at the peak. A third epoch sometimes helps and sometimes memorizes. The golden set decides, not the schedule.

Serving compressed models

A compressed model changes the serving math from Lesson 57. The weights are smaller, so more of them fit in GPU memory and the decode step reads them faster. That is the win. But the KV cache does not shrink with the weights. The notes the model takes while reading are still full size unless you quantize those too, and quantizing the KV cache is a separate choice with its own quality cost. I size the serving box with both numbers: weights plus KV cache for the concurrent requests I actually expect. I also check the kernels. A compressed model only runs fast if the serving engine has fast code for that precision. INT8 with no fast kernel can run slower than FP16 with a good one. I test the speed on the real box before I promise it.

When compression beats a bigger model

A smaller model that fits on one cheap GPU beats a bigger model that needs two expensive ones, if the quality holds. That is the common case. But there is a line. If the job needs rare knowledge, long reasoning, or judgment the small model never had, no amount of compression brings it back. Distillation cannot teach what the teacher barely knows. QAT cannot repair knowledge that was never in the student. When the golden set says the small model fails the hard cases no matter the settings, I stop shrinking and I keep the bigger model. Compression is a tool for the jobs where the big model was more than the job needed. It is not a way to make a small model smart.

What changes at 10x and 100x

At 10x the traffic, the serving box is the bottleneck, not the compression job. I compress once, then I spend my time on batching and the KV cache, because ten times the users means ten times the notes in GPU memory. At 10x the models, the compression pipeline is the bottleneck. I automate it: one config, one calibration set, one golden set per model, and the same eval gates on every run. At 100x, compression is a service with versions. Every compressed model carries its recipe: which precision, which prune, which teacher, which QAT epochs, and which golden set score it got. I can answer the question "which recipe made this model" a year later, because the answer is in the log, not in someone's memory.

Compression config snippet

This is the config for the careful shrink in this lesson. It is generic, not tied to one vendor's tool. The ideas are the same everywhere.

# compress.yaml — careful shrink of a 7B model for one 16 GB GPU box
base_model: support-assistant-7b
base_precision: fp16
base_size_gb: 14
target:
  max_weight_gb: 8
  min_golden_score: 78
quantization:
  method: awq          # protects the loud numbers, rounds the rest
  precision: int8      # 1 byte per number, 14 GB -> 7 GB
  calibration_prompts: 500   # real prompts from the logs, long ones included
pruning:
  method: structured   # whole pieces out, shape stays clean
  percent: 20          # 7 GB -> 5.6 GB
distillation:
  teacher: support-assistant-7b   # the big model, uncompressed
  enabled: true
  student: support-assistant-7b-int8
qat:
  epochs: 2
  learning_rate: 0.0001   # low on purpose, the rounding fights big steps
eval:
  golden_set: data/support_golden.jsonl
  split_by: [task, prompt_length]
serve:
  keep_base_model: true   # one click back until the small model survives a week

What the snippet means: the target comes first, weights under 8 GB and a golden score of 78 or better, and every step after it serves that target. AWQ protects the loud numbers before rounding. The calibration set is 500 real prompts, long ones included, because the calibration decides what the quantizer protects. Structured pruning keeps the shape clean so the GPU actually runs faster. Distillation teaches the student the teacher's style, not its facts. QAT runs two epochs at a low learning rate so the small numbers learn without chasing their own tail. The eval splits by task and by prompt length, because the overall score hides uneven damage. And the base model stays deployed behind a flag until the small one has survived a week.

One time it broke

This one stayed with me because the test said we were fine and the users said we were wrong. Both were right. The test was measuring the wrong prompts.

What broke

We shipped an INT4 model that scored 78 on the golden set, just above our floor. On Monday, the support team reported wrong answers on long customer messages. The model gave confident summaries of tickets it had clearly misread. It mixed up two customers in one long thread. It invented a refund step that did not exist. The app was healthy. The GPU was healthy. The latency dashboard looked great, because the model was small and fast. The quality dashboard said 78, because the golden set was all short questions.

Timeline

  • 09:05First ticket: the bot summarized a long thread and mixed up two customers. I try it myself with a short question. It answers fine. I try it with a long thread. It gets the names wrong.
  • 09:20I check the model and the serving path. Both healthy. Latency is down, memory is down, no errors in the logs. The model is doing what it was trained to do. That is the problem.
  • 09:40I read ten wrong answers side by side. All ten are long prompts, over 1,000 tokens. Every short answer in the same hour is fine. The damage has a shape, and the shape is length.
  • 10:05I open the calibration set. All 100 prompts are short, under 200 tokens. The quantizer never saw a long prompt, so it never protected the numbers that long prompts need.
  • 10:30I pull fifty real long prompts from the weekend and grade the answers by hand. The real score on long prompts is 52, not 78.
  • 11:00I roll the model back to the INT8 version for all users. The INT8 model is bigger and slower and it reads long threads correctly. That trade is correct.
  • 11:30I confirm from three long threads that the INT8 model is serving and the mixed-up summaries are gone.

What changed after

Three things. First, the calibration set is 500 real prompts, and a script checks the length mix before any quantize run starts. If the longest calibration prompt is shorter than the longest production prompt, the run stops. Second, the golden set is split by prompt length, and the long-prompt score has its own floor. An overall score of 80 with a long-prompt score of 52 does not ship. Third, every ship now includes a weekend of real long prompts graded by a human before the canary grows past five percent. The average score is a signal. The split score is the decision.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real compression job. Take a minute with it before you read my approach.

The prompt

Design model compression for an internal support assistant used by 200 engineers. The model is 7 billion parameters in FP16, 14 GB, scoring 82 on the golden set, writing at 25 ms per token. It must run on one 16 GB GPU box in the office. The target is a golden-set score of 78 or better and a p95 time to first token under 500 ms. Sketch the compression plan, the sizing, and the ship plan.

Clarifying questions to ask

  • How long are the real prompts: short questions, or long threads pasted in?
  • How many engineers ask at the same time: is 200 the total, or the Monday morning peak?
  • What does a wrong answer cost here: a wasted reply, or a bad command run against production?
  • What else shares the 16 GB box, and how much memory is really free for the model?
  • Who grades the answers, and how many can they read a week?

Rough sizing, worked

Start with the target. The box is 16 GB. The weights get 8 GB at most, because the KV cache needs the rest. INT8 halves the 14 GB model to 7 GB, which fits. A 20 percent structured prune takes it to 5.6 GB, which fits with room to spare. Speed follows the size: 8 plus 1.2 times 5.6 GB is about 15 ms per token, down from 25. Now size the KV cache, because the weights are only half the memory. At INT8 the KV cache costs about 0.25 MB per prompt token, from Lesson 57. A 512-token prompt holds about 128 MB of notes. The box has about 8 GB left after the weights, so it holds about 64 requests at once. If the Monday peak is 200 engineers asking at once, one box is not enough, and I say so now instead of finding out on Monday. I calibrate on 500 real prompts, long ones included. I distill from the big model for the style. I run two QAT epochs to repair the rounding. The expected score is 82 minus 2 for INT8, minus 4 for the prune, plus 2 for distillation, plus 2 for QAT, which is 80. That clears the floor of 78 with room for a bad day. The ship plan is a canary to one team first, then the whole org, with the big model one click away.

Trade-offs to name

INT8 buys half the size for about 2 quality points, and it is the cheapest trade on the table. INT4 buys another half for about 5 more points, and that is usually too much. Structured pruning buys speed and costs the numbers it removes, so I test per task. Distillation buys style and costs 90 minutes of job time. QAT buys back the damage and costs two hours. And the target is a split score, not an overall one. An overall 80 with a long-prompt 60 fails this design. The eval measures the proxy. The week of real users measures the thing.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 88 — How Models Are Trained
Coming next