Platform Engineering · Lesson 209

GPU and Model Formats for Inference

One question. One A100. One 7B model. The answer lands in 187 milliseconds, and I can tell you where every one of those milliseconds goes. Prefill reads the prompt and does the math. Decode writes the answer one token at a time and reads the whole model for each token. The two hops fail for different reasons, and the fix for one does nothing for the other. This lesson is the map of the GPU, the formats the model arrives in, and the harness that proves the numbers instead of trusting them.

Difficulty: Advanced · Know this first: Lesson 38, Lesson 180, Lesson 208

What you will build

By the end of this lesson you will have run a safetensors round trip on your own machine, built and run a tiny ONNX model, and run a repeatable benchmark harness against a local endpoint and read TTFT and TPOT off it. You will also have the roofline math to size a serving fleet: prefill time, decode time, KV cache size, and the GPU count for a request rate. The GPU parts run on your own hardware or your own cloud account. The code parts run here, on CPU, and their outputs are shown exactly as they ran.

What you need: Python 3 and pip for the local parts, and a GPU box or your own cloud account for the GPU parts. This machine has no GPU, so nothing in this lesson claims a measured GPU number. The GPU numbers are published specs plus roofline math, and the lesson shows the math. The code outputs are real runs on CPU and they are marked that way. Your numbers will differ. The shape will not.

The story: a file, a card, and a clock

A model is a file before it is a service. The file has a format. Safetensors, ONNX, TensorRT, GGUF. Each format is a different deal about what you give up and what you get. Then the file lands on a GPU, and the GPU has two numbers that matter: how fast it multiplies, and how fast it reads its own memory. Prefill needs the first. Decode needs the second. The table below is the whole lesson in numbers. The specs are the published NVIDIA numbers, checked on November 21, 2026.

GPUMemoryMemory bandwidthFP16 tensor, denseLink between GPUs
T416 GB320 GB/s65 TFLOPSPCIe Gen3
L424 GB300 GB/s121 TFLOPSPCIe Gen4 x16, 64 GB/s
RTX 409024 GB1,008 GB/sabout 83 TFLOPSPCIe Gen4 x16, 64 GB/s
A100 SXM 80GB80 GB2,039 GB/s312 TFLOPSNVLink 3, 600 GB/s
H100 SXM 80GB80 GB3,350 GB/s989 TFLOPSNVLink 4, 900 GB/s

Pick by memory first. A 7B model in FP16 is 14 GB, so it fits on all five. A 70B model in FP16 is 140 GB, so it fits on none of them alone. Pick by bandwidth second, because decode reads the whole model once per token. Pick by FLOPS third, because prefill is the only hop that uses them. A card with three times the FLOPS and the same bandwidth answers long outputs at the same speed. That is the mistake people make when they buy the bigger card.

The formats, in one line each. Safetensors is the safe way to store weights: a JSON header, raw tensors after it, no pickle, memory-mapped on load. ONNX is the portable graph: every framework exports to it, every runtime runs it, and the opset version is the thing that breaks. TensorRT is the fast engine: it compiles the model for one GPU and one driver, and it rebuilds when either changes. GGUF is the small file: quantized weights for running on a CPU or a small card, 4.2 GB for a 7B model at Q4_K_M against 14 GB in FP16.

One request walks through all of it now. A 7B model in BF16 on one A100 with 80 GB. One thousand twenty-four tokens in, sixteen out, batch of one. The FastAPI door from Lesson 38 takes the request. The format check reads the header. The weights sit warm in HBM. Prefill does the math. Decode writes the answer. The meter reads the trip.

Do this now, before the diagram

Write down the two numbers for your own card, if you have one. Memory bandwidth and FP16 TFLOPS. Then divide 14 GB by the bandwidth. That is your decode floor for a 7B model, in seconds per token. On an A100 it is 6.9 ms. On an H100 it is 4.2 ms. On a T4 it is 43.8 ms. If you do not have a card, use the table above and do the same division for the L4. The number you just computed is the speed limit for every long answer that card will ever write.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one question through the serving stack on one A100. These four paths are the same idea drawn four ways. One question, answered. One question, stopped by an engine built for the wrong GPU. One roofline trail, hop by hop. One afternoon at batch 32, mostly waiting on prefill. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the format check and never come back.

Path 1: the answered question

Metrics: skipped

One question walks the serving stack from the story. FastAPI door, tokenizer, format check, weights warm in HBM, prefill, decode. The answer is born at the decode hop, and the green dot starts back from there.

Client0 ms
FastAPI2 ms
Tokenizer3 ms
Format check5 ms
Weights7 ms
Prefill46 ms
Decode114 ms

Not run yet.

Path 2: the wrong-GPU engine

Weights: skipped

Same question, wrong card. The TensorRT engine was built for an H100 and the box holds an A100. The format check refuses the engine and the request stops there. The weights never load. The model never runs. The user gets an error instead of an answer.

Client0 ms
FastAPI2 ms
Tokenizer3 ms
Format checkrebuild fails
Weightsskipped

Not run yet.

Path 3: the roofline trail

Decode: the data hop

The same request, measured instead of timed. The prompt is 1,024 tokens. Prefill does 14.3 TFLOP of work at 312 TFLOPS and costs 46 ms. Decode reads 14 GB per token at 2,039 GB/s and costs 114 ms. The trail adds up to 187 ms, and the two big hops are the two different bottlenecks.

Prompt1,024 tokens
Prefill work14.3 TFLOP
Prefill time46 ms
Decode read14 GB/token
Decode time114 ms
Verdict187 ms

Not run yet.

Path 4: the batch of 32

Cold start: skipped

The same question at batch 32. Each hop shows its time. Prefill multiplies by 32 and costs 1,470 ms. Decode reads 30 GB a token now, weights plus KV cache, and costs 235 ms. The rest of the hops do not change. Batching is a prefill problem first and a decode problem second.

Client0 ms
FastAPI2 ms
Tokenizer3 ms
Prefill1,470 ms
Decode235 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one request through the serving stack, not a measurement from a live system, but the shape is honest. Prefill and decode are the big hops. They cost 46 ms and 114 ms at the simulator defaults, and the total is 187 ms. Switch the scenario and watch the bar change. In the cold story the weights hop grows to 3,800 ms, because the weights load off the disk. In the batch story the prefill hop grows to 1,470 ms, because thirty-two prompts are thirty-two times the work. Move a simulator slider and watch the prefill hop, the decode hop, and the total move with it.

Model size or batch: move the sliders and watch the roofline move

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the sizing question at the heart of this lesson: which hop is the bottleneck for your model, your precision, your batch, and your prompt. You set how big the model is, how many bytes each parameter costs, how many requests run at once, and how long the prompt is. Move a slider and watch the weights hop, the prefill hop, the decode hop, TTFT, TPOT, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: prefill follows FLOPS, decode follows bandwidth, and the model has to fit in memory first.

7B parameters

How many parameters the model has. 1B is a small model, 7B is the lesson default, 13B is a large one, 70B is the one that does not fit on one card in FP16.

FP16 (2 bytes/param)

How many bytes each parameter costs. FP16 costs 2, INT8 costs 1, FP4 costs 0.5. Smaller precision shrinks the weights and speeds up decode, and it charges accuracy.

1 request

How many requests run at once. Batching shares one copy of the weights, so throughput goes up. Prefill work multiplies by the batch, so TTFT goes up too.

1,024 tokens

How long the prompt is. Prefill work grows with the prompt, and the KV cache grows with it too. A long prompt is a slow prefill and a big KV cache.

Try this: (1) At the defaults, 7B in FP16, batch 1, 1,024 prompt tokens, the weights are 14.0 GB, the weights hop is 7 ms, prefill is 46 ms, decode is 114 ms, TTFT is 63 ms, TPOT is 7.1 ms, and the budget total is 187 ms. (2) Drop to 1B in FP16, batch 1, 256 prompt tokens. The weights fall to 2.0 GB, the weights hop to 1 ms, prefill to 2 ms, decode to 16 ms, TTFT to 13 ms, and the budget total to 39 ms. The small model is fast because there is less to read and less to multiply. (3) Push to 13B in FP16, batch 8, 2,048 prompt tokens. The weights are 26.0 GB, the weights hop is 13 ms, prefill is 1,365 ms, decode is 321 ms, the KV cache is 14.86 GB, TTFT is 1,388 ms, TPOT is 20.1 ms, and the budget total is 1,719 ms. The batch and the prompt did that, not the model. (4) Pick 70B in FP16. The weights are 140 GB and the status line says the model does not fit on one 80 GB A100. Switch to INT8 and it fits at 70 GB, with 10 GB left for the KV cache. Switch to FP4 and it fits at 35 GB.

Formulas, in plain terms, for one A100 at 312 TFLOPS and 2,039 GB/s. Weights GB = parameters times bytes per parameter. Weights hop = weights GB divided by 2,039 GB/s, in ms, rounded, minimum 1 ms. Prefill hop = 2 times parameters times prompt tokens times batch, divided by 312 TFLOPS, in ms, rounded. KV cache GB = batch times 0.5 MB per token at 7B, scaled by model size, times prompt tokens, divided by 1,024. Decode hop = 16 tokens times weights GB plus KV cache GB, divided by 2,039 GB/s, in ms, rounded. TTFT = 2 + 3 + 5 + weights hop + prefill hop. TPOT = decode hop divided by 16. Budget total = 0 + 2 + 3 + 5 + weights hop + prefill hop + decode hop + 4 + 6. Throughput = batch times 16 tokens, divided by decode seconds. Real cards run at 50 to 70 percent of published peak, so these are a floor, not a promise.

Lab: the format and the harness, on your own machine

Three steps, all run on CPU in this machine. No GPU here, so nothing below claims a GPU number. The format steps prove the file handling. The harness step proves the measurement method. The same harness against a GPU endpoint measures the real TTFT and TPOT.

Step 1: safetensors round trip

Save two tiny tensors, read the header back as JSON, and load them again. The header is the format: names, shapes, dtypes, and byte offsets, all in plain JSON. No pickle runs on load, so a file cannot run code when you open it. Then quantize one row to INT8 by hand and measure the error, so the size math in the story is not a claim. It is a measurement.

"""formats_demo.py: safetensors round trip + quantization size math.

Run with: PYTHONPATH=pylibs python3 formats_demo.py
safetensors 0.8.0, numpy 1.26.4. No GPU in this VM; the tensors are tiny
and the point is the format, not the model.
"""
import json
import os

import numpy as np
from safetensors.numpy import load_file, save_file

# Two tiny weight tensors, the shape a real layer would have, shrunk.
weights = {
    "layer0.weight": np.arange(12, dtype=np.float32).reshape(3, 4),
    "layer0.bias": np.array([0.5, -0.5, 0.25], dtype=np.float32),
}
save_file(weights, "/tmp/tiny.safetensors", metadata={"format": "pt"})
print("saved /tmp/tiny.safetensors:", os.path.getsize("/tmp/tiny.safetensors"), "bytes")

# The header is plain JSON: 8 length bytes, then the header, then raw data.
with open("/tmp/tiny.safetensors", "rb") as f:
    header_len = int.from_bytes(f.read(8), "little")
    header = json.loads(f.read(header_len))
print("header:", json.dumps(header, sort_keys=True))

back = load_file("/tmp/tiny.safetensors")
print("round trip exact:",
      np.array_equal(back["layer0.weight"], weights["layer0.weight"])
      and np.array_equal(back["layer0.bias"], weights["layer0.bias"]))
print("dtype on disk:", back["layer0.weight"].dtype)

# Fake INT8 quantization of one row: scale, round, dequantize, measure error.
row = np.array([0.12, -0.45, 0.9, -0.03, 0.66, -0.78, 0.21, 0.05],
               dtype=np.float32)
scale = np.abs(row).max() / 127.0
q = np.round(row / scale).astype(np.int8)
dq = q.astype(np.float32) * scale
print("int8 max abs error: %.4f (scale %.5f)" % (np.abs(row - dq).max(), scale))

# Size math for a 7B model. Bytes per parameter decide the file size.
for name, bpp in [("FP16", 2.0), ("INT8", 1.0), ("Q4_K_M", 0.6), ("Q4_0", 0.5)]:
    print("7B %s: %.2f GB" % (name, 7 * bpp))

Output, exactly as it ran (safetensors 0.8.0, numpy 1.26.4):

saved /tmp/tiny.safetensors: 236 bytes
header: {"__metadata__": {"format": "pt"}, "layer0.bias": {"data_offsets": [0, 12], "dtype": "F32", "shape": [3]}, "layer0.weight": {"data_offsets": [12, 60], "dtype": "F32", "shape": [3, 4]}}
round trip exact: True
dtype on disk: float32
int8 max abs error: 0.0035 (scale 0.00709)
7B FP16: 14.00 GB
7B INT8: 7.00 GB
7B Q4_K_M: 4.20 GB
7B Q4_0: 3.50 GB

The round trip is exact. The INT8 error on that row is 0.0035, and the file sizes are the ones in the story: 14.00 GB in FP16, 7.00 GB in INT8, 4.20 GB at Q4_K_M, 3.50 GB at Q4_0, for a 7B model.

Step 2: a tiny ONNX model, built and run

Build a two-op graph by hand, MatMul plus Relu, check it, save it, and run it with ONNX Runtime on CPU. This is the export-then-serve path in miniature. A real export from PyTorch produces the same shape: a graph, an opset version, and weights. The opset is the thing that breaks, so the script prints it.

"""onnx_demo.py: build a tiny ONNX model, check it, run it.

Run with: PYTHONPATH=pylibs python3 onnx_demo.py
onnx 1.23.2, onnxruntime 1.32.0. No GPU, no torch: the model is a
two-op graph (MatMul + Relu) built by hand, the same shape an export
produces. It proves the export-then-serve path works on CPU before
anyone trusts it on a GPU.
"""
import numpy as np
import onnx
from onnx import TensorProto, helper

# Graph: y = relu(x @ W + b). Inputs x (1x3), weights W (3x2), bias b (2).
W = helper.make_tensor("W", TensorProto.FLOAT, [3, 2],
                       [0.5, -0.5, 1.0, 0.0, -1.0, 2.0])
b = helper.make_tensor("b", TensorProto.FLOAT, [2], [0.1, -0.1])
mm = helper.make_node("MatMul", ["x", "W"], ["m"])
add = helper.make_node("Add", ["m", "b"], ["a"])
relu = helper.make_node("Relu", ["a"], ["y"])
graph = helper.make_graph(
    [mm, add, relu], "tiny",
    [helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 3])],
    [helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 2])],
    [W, b])
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 13)])
onnx.checker.check_model(model)
onnx.save(model, "/tmp/tiny.onnx")
print("saved /tmp/tiny.onnx, opset 13, nodes:",
      [n.op_type for n in model.graph.node])

import onnxruntime as ort
sess = ort.InferenceSession("/tmp/tiny.onnx", providers=["CPUExecutionProvider"])
x = np.array([[1.0, 2.0, -1.0]], dtype=np.float32)
print("input:", x.tolist(), "-> output:",
      sess.run(None, {"x": x})[0].tolist())
print("numpy check:", np.maximum(x @ np.array(
    [[0.5, -0.5], [1.0, 0.0], [-1.0, 2.0]], dtype=np.float32)
    + np.array([0.1, -0.1], dtype=np.float32), 0).tolist())

Output, exactly as it ran (onnx 1.23.2, onnxruntime 1.32.0):

saved /tmp/tiny.onnx, opset 13, nodes: ['MatMul', 'Add', 'Relu']
input: [[1.0, 2.0, -1.0]] -> output: [[3.5999999046325684, 0.0]]
numpy check: [[3.5999999046325684, 0.0]]

The runtime output matches the numpy check exactly. The model ran, and the check proves it ran the graph you wrote.

Step 3: the benchmark harness

The server below is a stand-in, not a model. For each output token it does one real 256x256 matmul on CPU and streams the token back. The harness sends 20 identical requests, one at a time, after 2 warmup requests, and reports TTFT, TPOT, p50 and p95 total latency, and throughput. Same harness, same flags, every run. That is what makes two numbers comparable. Run it three times and keep the middle run. A single run is noise.

"""bench_server.py: a tiny CPU inference stand-in for the bench harness.

Run with: python3 bench_server.py  (stdlib only, no GPU, no model)
Listens on 127.0.0.1:8471. POST /generate with JSON
{"prompt_tokens": 1024, "max_tokens": 16}. For each output token it does
one real 256x256 matmul (real CPU work, a few ms) and streams the token
back with chunked encoding. The point is the harness, not the model:
on a GPU box the same bench.py measures real TTFT and TPOT.
"""
import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import numpy as np

A = np.random.RandomState(0).rand(256, 256).astype(np.float32)


class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def do_POST(self):
        if self.path != "/generate":
            self.send_response(404)
            self.end_headers()
            return
        n = int(self.headers.get("Content-Length", "0"))
        body = json.loads(self.rfile.read(n) or b"{}")
        max_tokens = int(body.get("max_tokens", 16))
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Transfer-Encoding", "chunked")
        self.end_headers()
        for i in range(max_tokens):
            _ = A @ A  # real work: one matmul per token, CPU
            chunk = ("token-%d\n" % i).encode()
            self.wfile.write(b"%x\r\n%s\r\n" % (len(chunk), chunk))
            self.wfile.flush()
        self.wfile.write(b"0\r\n\r\n")
        self.wfile.flush()


if __name__ == "__main__":
    print("listening on 127.0.0.1:8471", flush=True)
    ThreadingHTTPServer(("127.0.0.1", 8471), Handler).serve_forever()
"""bench.py: the repeatable inference benchmark harness.

Run with: python3 bench.py  (server must be running: bench_server.py)
Sends 20 identical requests, one at a time, and reports TTFT (time to
first token), TPOT (time per output token after the first), p50/p95 total
latency, and throughput. Same harness, same flags, every run: that is
what makes two numbers comparable. Run it three times and keep the
middle run. A single run is noise.
"""
import http.client
import json
import statistics
import time

N = 20
BODY = json.dumps({"prompt_tokens": 1024, "max_tokens": 16})


def one_request():
    conn = http.client.HTTPConnection("127.0.0.1", 8471, timeout=30)
    t0 = time.perf_counter()
    conn.request("POST", "/generate", BODY,
                 {"Content-Type": "application/json"})
    resp = conn.getresponse()
    first = resp.readline()  # first token arrives here: TTFT
    ttft = (time.perf_counter() - t0) * 1000
    tokens = 1
    while resp.readline():
        tokens += 1
    total = (time.perf_counter() - t0) * 1000
    conn.close()
    tpot = (total - ttft) / (tokens - 1) if tokens > 1 else 0.0
    return ttft, tpot, total, tokens


rows = []
for _ in range(2):  # warmup: the first requests pay the cold-start cost
    one_request()
rows = [one_request() for _ in range(N)]
ttfts = sorted(r[0] for r in rows)
tpots = sorted(r[1] for r in rows)
totals = sorted(r[2] for r in rows)
toks = sum(r[3] for r in rows)
wall = sum(r[2] for r in rows) / 1000.0


def pct(sorted_list, p):
    i = min(len(sorted_list) - 1, int(p * len(sorted_list)))
    return sorted_list[i]


print("requests: %d, tokens per request: %d" % (N, rows[0][3]))
print("TTFT  median %.1f ms, p95 %.1f ms" % (statistics.median(ttfts), pct(ttfts, 0.95)))
print("TPOT  median %.1f ms, p95 %.1f ms" % (statistics.median(tpots), pct(tpots, 0.95)))
print("total median %.1f ms, p95 %.1f ms" % (statistics.median(totals), pct(totals, 0.95)))
print("throughput %.1f tok/s (%d tokens in %.2f s)"
      % (toks / wall, toks, wall))

Output, the middle of three runs against the local CPU endpoint in this machine:

requests: 20, tokens per request: 16
TTFT  median 13.9 ms, p95 52.7 ms
TPOT  median 12.7 ms, p95 20.7 ms
total median 207.5 ms, p95 343.8 ms
throughput 72.0 tok/s (320 tokens in 4.44 s)

The three runs measured 60.8, 72.0, and 75.8 tokens a second. That spread is the point. One run says 60.8 and another says 75.8, and both ran the same code on the same box. The middle run is the number to quote, and the spread is the number to remember. On a GPU box the same harness measures real TTFT and TPOT, and the same rule applies: warmup, twenty requests, three runs, keep the middle.

Do this: run all three steps on your own machine, then run the harness against your own GPU endpoint with the same flags. Write down TTFT and TPOT. Then change one thing, the batch size or the precision, and run it again. The difference between the two runs is the only number that matters.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Serving a model on a GPU is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many GPU names you can recite. They are about whether the numbers change what gets built. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · One GPU, one format, read the meter

At the basic level you can name the GPU, the format, and the two numbers that matter. You run nvidia-smi, you read memory used and free, and you can say whether a model fits. A 7B model in FP16 is 14 GB. An 80 GB A100 holds it with room for the KV cache. A 16 GB T4 holds it with almost nothing left. The failure that ends this level is the first model that does not fit. You learn the fit math the day a deploy fails on it.

Types:

  • GPU memory: weights, the KV cache, and activations must all fit.
  • Precision: FP32 (4 bytes per parameter), FP16 and BF16 (2), INT8 (1), INT4 (0.5).
  • Model formats: PyTorch checkpoints, safetensors, ONNX, GGUF, TensorRT engines.
  • nvidia-smi: memory used, utilisation, temperature, power.

Uses: Choosing hardware and a format before deploying a model, so it fits and runs at the speed the product needs.

Scale and failure: Weights are only the start: the KV cache grows with batch size and context length, so a model that fits at batch 1 can run out of memory at batch 16. Plan memory for the peak batch and the longest context you allow.

Trade-off: Lower precision fits bigger models on smaller cards and costs some accuracy; check the evaluation set at each precision.

Do this: run nvidia-smi on your box and write down memory used and free. Compute the weights size of a 7B FP16 model (14 GB) and a 13B FP16 model (26 GB) and check both against the free memory. Then run step 1 of the lab and read the safetensors header with your own eyes.

Try it: Basic

I have a 7B model in FP16 and a 16 GB T4. Does it fit?

Now do it: On your GPU box, run nvidia-smi, write down memory used and free, and check the 14 GB for a 7B FP16 model and the 26 GB for a 13B FP16 model against the free memory.

Intermediate · The roofline, done by hand

At the intermediate level you do the roofline math before you pick a card. Prefill is two times parameters times prompt tokens, divided by TFLOPS. Decode is sixteen tokens times weights plus KV cache, divided by bandwidth. You run the numbers for your model, your prompt, and your batch, and you know which hop is the bottleneck before the first request lands. You also export a model to ONNX and run it, so the format is not a name on a slide.

Types:

  • Prefill: processing the prompt, limited by compute (TFLOPS).
  • Decode: generating tokens one at a time, limited by memory bandwidth.
  • Time to first token (TTFT) and inter-token latency.
  • Export paths: ONNX Runtime, TensorRT, vLLM-ready weights.

Uses: Predicting latency and choosing the right GPU for a workload before buying or renting it.

Scale and failure: Long prompts make prefill dominate and favour high-compute cards; long answers make decode dominate and favour high-bandwidth cards. Batching raises throughput and raises each request's latency.

Trade-off: Bigger batches use the GPU better and make each user wait longer. The SLO decides where the batch size stops.

Do this: take the simulator defaults and change one control at a time. Double the prompt and watch prefill double. Double the batch and watch prefill double again. Then run step 2 of the lab and run the tiny ONNX model.

Try it: Intermediate

I double the prompt. What happens to prefill?

Now do it: Run the drill above: take the simulator defaults, change one control at a time, double the prompt and watch prefill double, then run step 2 of the lab and run the tiny ONNX model.

Advanced · Multi-GPU, the engine build, and the KV budget

At the advanced level you size the KV cache and you build engines per card. A 70B model in FP16 is 140 GB, so it needs two A100s or one smaller precision. Tensor parallel splits the layers across the cards, and NVLink carries the activations at 600 GB/s on A100 and 900 GB/s on H100, against 64 GB/s for PCIe Gen4. The KV cache gets a budget of its own: batch 8 at 4,096 tokens is 16 GB for a 7B model, and that memory comes out of the same 80 GB as the weights. The engine is built in CI for each GPU, never on the serving box.

Types:

  • Tensor parallelism: splitting each layer across GPUs.
  • Pipeline parallelism: splitting layers into stages on different GPUs.
  • Interconnects: NVLink between GPUs versus PCIe.
  • KV cache budgeting and PagedAttention.

Uses: Serving models too large for one GPU, and serving many concurrent users efficiently.

Scale and failure: Tensor parallelism needs fast interconnects; across PCIe or between machines, communication can cost more than the computation saved. PagedAttention reduces wasted KV cache memory so more requests fit on the same card.

Trade-off: More GPUs per model lower latency for big models and raise cost and failure surface; quantising to fit one GPU is cheaper and costs accuracy.

Do this: size the KV cache for batch 8 at 4,096 tokens on a 7B model (16 GB), then check what is left of the 80 GB after the 14 GB of weights. Then run the bench harness from step 3 of the lab and read TTFT and TPOT off it.

Try it: Advanced

My 70B model in FP16 will not fit on one 80 GB A100. What are my options?

Now do it: On your GPU box, run the drill above: size the KV cache for batch 8 at 4,096 tokens on a 7B model (16 GB), check what is left of the 80 GB after the 14 GB of weights, then run the bench harness from step 3 and read TTFT and TPOT off it.

Master · The harness in CI, and the alerts that follow

At the master level the benchmark harness runs in CI and the budget is a test. Every model change runs bench.py with the same flags, and a p95 TTFT that crosses the budget fails the build. The harness output feeds the drift and eval alerts in Lesson 216, so a slow model and a wrong model trip the same wire. PagedAttention tuning lives here as overflow: Lesson 181 teaches the mechanism, and tuning its block size for your workload is the Master work that follows this lesson, not a requirement of it. The team still reads the dashboard every week. The dashboard reads the harness, not a guess.

Types:

  • Benchmark harnesses run in CI on every model or engine change.
  • Latency budgets per percentile: p95 TTFT, p95 inter-token latency.
  • Capacity models: requests per second per GPU at the SLO.
  • Cost per million tokens, tracked over time.

Uses: Inference platforms where model, engine, and hardware change often, and every change must be priced and checked.

Scale and failure: Engine and driver upgrades change performance in both directions; without a benchmark in CI, a regression ships unnoticed. Capacity is measured as throughput at the latency SLO, not peak throughput.

Trade-off: Continuous benchmarking costs GPU time on every change and catches regressions before customers do.

Do this: wire bench.py into CI with a TTFT budget of your own, and run it on every model change. Then pick one number from the dashboard, TTFT or TPOT, and write the alert that fires when its p95 crosses budget.

Try it: Master

A model change makes p95 TTFT cross the budget. What happens?

Now do it: On your GPU box, run the drill above: wire bench.py into CI with a TTFT budget of your own on every model change, then pick TTFT or TPOT and write the alert that fires when its p95 crosses budget.

Go deeper

The roofline, in full

Prefill is compute. Two FLOPs per parameter per prompt token, times the batch, divided by the card's TFLOPS. Decode is memory. Every output token reads the weights once and the KV cache once, divided by the card's bandwidth. The two formulas are the whole sizing method, and they are the formulas in the simulator. Real cards run at 50 to 70 percent of their published peak, so treat the roofline numbers as a floor, not a promise. A prefill that the math says costs 46 ms will cost 46 ms or more on a real card, never less. The same request on an H100 costs 14 ms of prefill and 69 ms of decode, with TTFT of 28 ms, because the H100 has 989 TFLOPS and 3,350 GB/s against the A100's 312 and 2,039. The H100 is about three times faster at prefill and about 1.6 times faster at decode. The two speedups are different numbers because the two hops are different bottlenecks.

Multi-GPU: tensor parallel, pipeline parallel, and the link between cards

A model that does not fit on one card needs two or more, and there are two ways to split it. Tensor parallel splits each layer across the cards, so every layer needs an all-reduce across the link. Pipeline parallel splits the layers into stages, so each card holds whole layers and the activations flow from one stage to the next. Tensor parallel needs the fast link. NVLink moves 600 GB/s on A100 and 900 GB/s on H100. PCIe Gen4 x16 moves 64 GB/s, and Gen5 moves 128 GB/s. That is why tensor parallel across PCIe cards is slow: the link is ten times slower than NVLink, and the all-reduce happens once per layer per token. A 70B model in FP16 needs two A100s at minimum, 140 GB against 160 GB of two cards, and almost nothing is left for the KV cache. In INT8 it needs one card, 70 GB, with 10 GB left. The precision decision and the GPU count decision are the same decision.

The formats, as serving formats

Safetensors is the storage format. A JSON header with names, shapes, and dtypes, then raw tensors. No pickle, so loading cannot run code from the file. Memory-mapped on load, so the operating system pages the weights in instead of copying them. ONNX is the portable format. A graph of operators, an opset version, and a runtime on every platform. The opset is the thing that breaks: export with opset 13 and run with a runtime that expects 17, and the load fails on an operator that does not exist yet. TensorRT is the compiled format. It takes the ONNX graph and builds an engine for one GPU and one driver, with FP16 and INT8 and FP8 kernels picked for that card. The engine file is not portable. Build it per GPU in CI and ship the engine with the GPU label on it. GGUF is the small format. Quantized weights for a CPU or a small card, with names that say the precision: Q8_0 is 8 bits, Q4_K_M is about 4.85 bits and 4.2 GB for a 7B model, Q4_0 is 4 bits and 3.5 GB. The device-scale intro lives in Lesson 208, which teaches TFLite, ONNX, and Core ML at device scale. This lesson is the server side: export, serve, measure the accuracy cost.

One TensorRT command to know, run on your own GPU box, not here. This machine has no GPU, so no output is shown for it, and no output is claimed.

trtexec --onnx=model.onnx --saveEngine=model_h100.engine --fp16
# Build one engine per GPU. The file name carries the card.
# Move the engine to another card and it rebuilds on first load.

The accuracy cost, measured

Quantization buys memory and speed and charges accuracy. The charge must be measured, not assumed. Run the same eval set on the FP16 model and the quantized model, and split the results by prompt length, because long prompts drift first. If the INT8 model scores the same on short prompts and worse on long ones, the fix is not always to go back to FP16. Keep the weights in INT8 and the KV cache in FP16, or keep the long-prompt traffic on the FP16 deployment and the short traffic on INT8. Two deployments, one router, and the router is the FastAPI door from Lesson 38. The eval harness for the accuracy side lives in the eval lessons, and the benchmark harness in this lesson is the latency side. Both numbers ship together or neither ships.

Inference observability: the five graphs

Five graphs, one per question. TTFT answers whether the first token is fast. TPOT answers whether the tokens keep coming. Queue depth answers whether the requests are waiting for a free slot. GPU memory used answers whether the KV cache is about to evict. SM utilization answers whether the tensor cores are working or waiting on memory. Low SM utilization with high memory bandwidth is a healthy decode. Low SM utilization with low memory bandwidth is a starved GPU, and the fix is a bigger batch, not a bigger card. The harness in this lesson produces the first two numbers. The fold-in in Lesson 216 wires the harness output into drift and eval alerts. That is the apply step. This lesson is the teach step.

What changes at 10x and 100x

At 10x the requests, the batch grows and TTFT grows with it, so the TTFT budget is the first thing that breaks. The fix is more replicas, not a bigger batch, because a bigger batch is what broke TTFT. At 10x the context length, the KV cache grows tenfold, and a 7B model at 32,768 tokens needs 16 GB per sequence. The fix is PagedAttention, which Lesson 181 teaches, and a KV budget per deployment. At 100x, the fleet is the product. The harness runs in CI, the engines are built per GPU in CI, and the autoscaler scales on TTFT and queue depth, not on CPU. The hands-on for the serving engines themselves, TGI, Triton, TensorRT-LLM, the OpenAI-compatible client, and inference autoscaling, lives in the Lesson 181 fold-ins. This lesson is the GPU and the format underneath them.

Further reading

The device-scale formats intro: Lesson 208 On-Device AI. The serving front door: Lesson 38 FastAPI. The serving engines: Lesson 181 vLLM and Serving Engines. The eval side of the accuracy cost: Lesson 180 LLM Inference at Scale. Per-cloud GPU instances: Lesson 218 AI on AWS, Lesson 219 AI on Azure, Lesson 220 AI on GCP. NVIDIA L4 product page, with the published specs table. A100 vs H100 specs and pricing comparison, checked November 21, 2026.

One time it broke

What broke

A team moved its serving fleet from H100 boxes to A100 boxes to save money. The model was the same. The weights file was the same. The TensorRT engine file was the same, because nobody knew the engine was built for a card. Every pod rebuilt its engine on first request, 850 ms, and loaded its weights cold off the disk, 3,800 ms. The first request per pod took 4,825 ms. TTFT went from 28 ms to 4,825 ms for that request, and the rollout restarted pods all day, so the cold start kept happening all day.

Timeline

  • 09:00The new fleet rolls out. Pods start on A100 boxes with H100 engines.
  • 09:05The first user request per pod takes 4,825 ms. The dashboard shows p99 TTFT at 4,825 ms and median TTFT at 63 ms.
  • 09:20The team blames the model. The model did not change. I blame the network. The network did not change.
  • 10:15I open a pod log. The first line says the engine is rebuilding. The second line says the weights are loading from disk.
  • 10:40I check the engine build stamp. It names the H100. The box under it is an A100.
  • 11:00The engines are rebuilt for A100 in CI, the warmup request runs after every deploy, and the cold start is gone.

What changed after

Three things. First, the engine is built per GPU in CI, and the engine artifact carries the GPU label. A deploy to a card without a matching engine fails the build, not the first request. Second, a warmup request runs after every deploy, so the first real user never pays the cold start. Third, p95 TTFT has an alert, because median TTFT hid this failure for an hour. The median was 63 ms the whole time. The p99 was 4,825 ms. The median is not the user experience.

If you remember nothing else

Design exercise

Size the serving fleet. A 7B model in BF16 on A100s. Twenty requests a second, 1,024 tokens in, 128 out. TTFT must stay under 200 ms. How many A100s, and what batch size? Name the trade-off you are making.

Clarifying questions to ask

  • What is the TTFT budget, and is it p50 or p95? A p95 budget is a different fleet from a p50 budget.
  • What is the output length distribution, not just the average? A few 1,000-token answers change the decode math completely.
  • Does the traffic arrive steady or in bursts? Bursts need headroom, and headroom is idle cards.
  • What happens to TTFT when the batch doubles? If nobody knows, the sizing is a guess.
  • Is a second of latency acceptable for any request, or does the SLO cover all of them? The answer decides the batch size.

Worked approach

Start with one request. Prefill is 46 ms and TTFT is 63 ms, so one request at a time meets the budget. But one request at a time serves 1.05 requests a second, because decode takes 910 ms for 128 tokens. Batching is the only way to go faster.

Then the batch that fits the budget. At batch 2, prefill is 92 ms and TTFT is 109 ms, inside the budget, and one GPU serves 1.93 requests a second. Twenty divided by 1.93 is 10.4, so 11 GPUs. At batch 8, prefill is 368 ms and TTFT is 385 ms, outside the budget, and one GPU serves 5.34 requests a second, so 4 GPUs.

Then the trade-off. Hold the 200 ms TTFT budget and buy 11 GPUs, or relax TTFT to 400 ms and buy 4. The third option is the H100. Prefill on H100 is 14 ms at batch 1, so batch 8 costs about 116 ms of prefill and TTFT stays under 200 ms with 4 cards. The answer is not a number. The answer is the budget, the batch, and the card, priced together.

Trade-offs to name. TTFT against GPU count. Batch size against latency. Card price against card speed. A cheaper card that needs 11 replicas is not cheaper than a dearer card that needs 4. Say which SLO is the hard one before you size anything, because the sizing changes completely when it moves.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 210
Sequence and Vision Networks: CNN, RNN, LSTM, and Jetson

Lesson 185 built the machine: layers, a bend, a loss, a gradient, a loop. This lesson gives that machine three shapes of data and one place to live. A picture is a grid, and a convolution reads it by sliding a small window across…

Go to Lesson 210 →

Or browse the full course from the Lessons index.