You type a question into a chat box and hit send. The answer appears word by word, like someone typing it live. That is LLM inference. Inference is the name for running a trained language model to answer a question. The model already learned everything it knows during training. Inference is the part where it uses that knowledge, live, for your question. It exists because a model that only trained and never answered would be useless. I run small models on my own machine with Ollama for my k8s-ai-sre project, and the same shape holds at every size: the model reads your question, then it writes the answer one piece at a time. This lesson follows one question through that trip, counts where the time goes, and shows you the levers I actually pull when the answers get slow or the GPUs run out of memory.
Every hop charges rent in milliseconds. The bar below is a worked example for one answer of 60 tokens, not a measurement from a live system, but the shape is honest: the writing costs far more than the reading on a short question, and the reading takes over when the question gets long. The answer is 60 tokens in every scenario, so the decode hop is the same size in the normal and long scenarios. What changes is the queue, the prefill, and the memory. Step through the lesson or switch the scenario and watch the bar change. The rush hour nearly doubles the trip without the model doing anything different. The long prompt adds three quarters of a second of reading before a single word is written.
Extra picture · starts simple, builds to the full picture
Starts from scratch with one user and one GPU doing prefill then decode; adds the queue, scheduler and batch, then the KV cache; ends with prefill and decode split onto separate GPU sets with KV transfer between them (Master: disaggregated serving at scale). I read the picture first. The words under each step are only labels.
where the moving unit is nowalready visitednot reached yetRequest · the way thereResponse · the way back
Extra picture · starts simple, builds to the full picture
Starts from scratch with plain one-token-at-a-time decode; adds the draft-model shortcut; ends with the preemption branch and its user-visible cost (Master: serving as a contract - p99 and isolation). I read the picture first. The words under each step are only labels.
where the moving unit is nowalready visitednot reached yeta step where something stops or breaksRequest · the way thereResponse · the way backFailure · the packet stops here
The budget above is three fixed stories. This box is yours. Make the prompt longer, grow the batch, empty the prefix cache, and watch the first word get later and the answer get slower. Then ask the question I ask before every serving change: which number moved, and is that the number my users feel?
How long the question is. Prefill time is prompt tokens x 0.35 ms. At 512 tokens that is about 180 ms. At 4,096 tokens it is about 1,434 ms. The KV memory grows with it too.
How many requests share the GPU at once. Time per token is 25 ms plus 1 ms for each request in the batch above 8. At a batch of 4 that is 25 ms per token. At a batch of 32 it is 49 ms per token. Decode time is 60 tokens x time per token.
How much of the prefill work is skipped. Effective prefill is prefill time x (1 minus hit rate). At 60 percent hit, only 40 percent of the prefill work runs, so 180 ms becomes 72 ms. At 100 percent the prefill is skipped entirely.
INT8 does the math with smaller numbers, so prefill time and per-token decode time are multiplied by 0.6, and KV memory per token is halved, from 0.5 MB to 0.25 MB. It does not change the queue, the tokenizing, or the streaming.
Try this: (1) Set the prompt to 4,096 tokens, the batch to 32, and the cache hit rate to 0, still on FP16. The prefill is about 1,434 ms, each token costs 49 ms, the decode total is 2,940 ms, the time to first token is 1,469 ms, and the full response is 4,459 ms. (2) Now set the prompt to 128, the batch to 1, the cache hit rate to 100, and switch to INT8. The prefill is skipped, each token costs 15 ms, the decode total is 900 ms, the time to first token is 35 ms, and the full response is 985 ms. Same model, same question shape. Only the inputs changed.
Three questions. No score is saved anywhere — this is just you against the material.
I read this topic four times here. First as one user waiting on one answer, then as someone sharing the GPU, then as someone attacking the slow part on purpose, then as the person who owns the serving contract.
Definition: Inference is the model doing its job. A prompt goes in, an answer comes out, one token at a time. Two phases. Prefill reads the whole prompt at once. Decode writes the answer one token at a time. Prefill is fast math. Decode is the slow part, because each new token waits for the last one.
Types:
Uses: I use the basic level when I only need an answer and I am the only user. I ask my local model to explain a log. I wait a few seconds. It does not matter if it is slow, because nobody else is waiting.
Dimensions: At one user, the only limit is patience. What breaks is the wait: a long answer feels slow because decode is one token at a time. I notice it by watching the words appear, or not appear. The cost is a few seconds of my time, or a few cents on a hosted service. What I give up is speed. A single request on a single GPU is the most expensive way to run a model per token, and at this level I accept that.
Definition: At this level I stop running one request at a time. Batching runs many requests on the same GPU at once. The KV cache is the model's notes from reading a prompt, kept so the next token does not have to re-read everything.
Types:
Uses: I use this level when more than one person uses the model. A support bot gets ten questions at once. Continuous batching keeps the GPU busy and the queue short. I turn the prefix cache on when many prompts share the same instructions.
Dimensions: As load grows, throughput goes up and latency per request goes up too. That is the trade, and the batch size dial controls it. What breaks is the KV memory. The cache fills, the engine evicts notes mid-answer, and the user sees a long pause in the middle of a sentence. I notice it on cache memory used and on time per token climbing. The cost is complexity: a scheduler that decides every token step instead of once per group. What I give up is predictable latency for a single request, in exchange for more requests per GPU.
Definition: At this level I attack the decode bottleneck directly. Decode is one token at a time, so I either guess ahead, split the work, or do the math with smaller numbers.
Types:
Uses: I use this level when decode is the bottleneck and the p99 time per token is the complaint. On easy text the small model guesses right most of the time and the win is big. I split prefill from decode when long prompts keep freezing thirty other users' answers.
Dimensions: As load grows, the wins shrink or grow depending on the traffic. Speculation wins less on hard text. Disaggregation costs a network hop and a KV transfer. What breaks is the guess rate. I notice it when accepted tokens per step falls and the speedup disappears. The cost is engineering time and a second model to maintain, or a second fleet of GPUs. What I give up is simplicity for speed, and on some prompts I give up a little answer quality to quantization, so I test on my real prompts, not a benchmark.
Two numbers describe an answer, not one. Time to first token is how long the user waits for the first word. Time per output token is how fast the rest arrives. A long prompt makes the first number big and the second number small. I measure both, because a user who waits four seconds for the first word has already decided the service is slow, even if the rest streams fast.
The fix for a slow first token is not a bigger GPU. It is a shorter prompt, or a cached prefix. I keep the system prompt and the runbook pages in the prefix cache, so the prefill only pays for the new question. I prove it by timing the same question twice. The second call must be faster than the first.
Definition: At this level inference is a service with a contract. Time to first token has its own alert. Time per token has its own alert. The budget is a number, and the load test proves it before the traffic arrives.
Types:
Uses: I use this level when the model is in the path of real users and a slow model is a page. I set the alerts, I run the load test from Lesson 35, and I know which hop owns which share of the budget before the traffic shows up.
Dimensions: At 10x the queue breaks first, then the KV memory. I add GPUs, cap prompt length, and turn on the prefix cache. At 100x the design changes: serving capacity near the users, prefill split from decode, and the cache as its own service. What breaks at this scale is the thing with no alert on it, so every number has one. The cost is real money and real on-call time. What I give up is the ability to change things casually. A serving change is a deploy, with a load test and a rollback plan.
Averages hide the outage. The mean latency can look fine while one user in twenty waits ten seconds. I alert on p99 time to first token, not on the mean, and I set the alert from the last week of real traffic, not from a guess.
When the KV cache fills, the engine must choose. It can queue new requests, or it can preempt a running one and recompute it later. Queuing is honest. Preemption is expensive. I pick queuing with a clear error after a fixed wait, because a request that waits forever is worse than a request that fails fast and retries. I test it by filling the cache on purpose and watching the queue behave.
The lesson above is one question through nine hops. Production is the rest of it. Here is what I actually spend my time on once the budget is written down and the first dashboard is up.
A small fast model guesses the next few tokens, and the big model checks them all at once. If the guesses are right, the big model accepts them and the answer moves faster than one token at a time. If a guess is wrong, the big model corrects it and the small model tries again. This works because checking tokens is cheap and writing them one by one is the bottleneck. It does not change what the answer says. The big model still decides. It just decides several tokens per step instead of one. I use it when decode is the bottleneck and the small model is good enough to guess right most of the time. On easy text it wins big. On hard text the guesses miss and the win shrinks.
Prefill and decode want different things. Prefill wants raw compute to read a long prompt fast. Decode wants fast memory reads and low latency per token. When they share one GPU, a long prefill blocks every decode in progress, and every user watching words appear sees a pause. Disaggregated serving splits them onto different GPUs: one set of machines reads prompts, another set writes answers, and the KV notes travel between them over the network. The split costs a network hop and a transfer of the KV cache. It buys isolation. A 4,000-token prompt can no longer freeze the decode of thirty other users. I reach for this when long prompts are common and the p99 time to first token keeps getting blamed on the wrong hop.
Old batching waited for a whole group of requests to finish before starting the next group. That wastes the GPU, because short answers finish early and their slots sit empty until the longest answer in the group is done. Continuous batching fills each slot the moment it frees up. A new request joins the running batch mid-flight instead of waiting for the next round. Throughput goes up and queue time goes down, and the cost is a scheduler that has to make a decision every single token step instead of once per group. Every serious serving setup I have seen uses this. If the scheduler is still static batching, that is the first upgrade I ask about.
Quantization is doing the model math with smaller numbers. FP16 uses 16 bits per number. INT8 uses 8 bits, and FP8 uses 8 bits in a floating format. Smaller numbers mean the weights take half the memory, the math runs faster, and the KV notes take half the space. The price is precision. The model rounds more, and on some prompts the answer gets a little worse. The mistake is to assume the loss is the same everywhere. I test quantization on the real prompts the service actually gets, not on a benchmark. A support chatbot and a code assistant fail in different places. If the quality holds on my prompts, I take the speed. If it does not, I quantize only the parts that are safe, or I stay on FP16 and buy more GPUs.
GPU memory is finite and the KV cache is hungry. When the cache fills up, something has to go. Eviction picks which notes to throw away, usually the oldest or the least recently used. Paging moves notes to CPU memory instead of throwing them away, so a request can resume later at the cost of a slow copy back. Both are the same idea as cache eviction from Lesson 07, except the stakes are higher: evicting the wrong request mid-answer means re-reading its whole prompt, and the user sees a long pause in the middle of their answer. I set a hard cap on KV memory per request, alert at 80 percent of the total, and I never let one request take the whole GPU.
At 10x the traffic, the queue is the first thing that breaks, then the KV memory. I add GPUs, I cap prompt length, I turn on the prefix cache, and I watch the queue depth like a hawk. The model itself is rarely the problem at 10x. At 100x, the design changes. One region is not enough, so I put serving capacity near the users, I split prefill from decode, and I treat the prefix cache as a first-class service with its own capacity and its own alerts. The budget stops being a target and becomes a contract, and prefill and decode own separate shares of it. The load test from Lesson 35 is how I prove the contract before the traffic arrives, not after.
A short serving config for one replica. These are the knobs I actually set before I touch the model:
# serving.yaml — one replica of the chat service model: chat-assistant-v3 precision: fp16 # int8 when quality tests pass on real prompts max_prompt_tokens: 4096 # reject longer prompts with a clear error, do not queue them max_batch_size: 32 # above this, per-token latency hurts more than throughput helps max_kv_memory_gb: 60 # hard cap: the KV cache never takes more than this prefix_cache: on # reuse notes for identical prompt beginnings max_queue_wait_ms: 2000 # waiting longer than this, shed load instead of queueing forever
What the snippet means: prompts longer than 4,096 tokens get a clear error instead of sitting in the queue and blocking everyone behind them. The batch is capped at 32 because past that point each token gets slower faster than throughput grows. The KV cache gets a hard cap of 60 GB so one long prompt can never fill the GPU. The prefix cache is on because the system instruction is the same on every request. And a request that would wait more than 2 seconds in the queue is turned away with a retry signal instead of making the user stare at a spinner. Those six lines are the difference between a slow day and an outage.
Four requests arrive together. One of them has a 4,000 ms prefill. Without chunking, the short requests wait behind it. With chunked prefill, the long job yields every 500 ms and the short ones start sooner.
Run it with python3 queue_two_ways.py. The numbers are the whole point, so read them before you trust the paragraph.
python3 queue_two_ways.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. queue_two_ways.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.
Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.
python3 --version python3 -u queue_two_ways.py python3 -m json.tool --help
Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.
def queue_two_ways():
jobs = [("short-1", 200), ("long", 4000), ("short-2", 200), ("short-3", 200)]
t, waits = 0, {}
for name, prefill in jobs: # one prefill at a time, no chunking
waits[name] = t
t += prefill
print("no chunking, wait before prefill starts (ms):", waits)
assert waits["short-2"] == 4200 and waits["short-3"] == 4400
# chunked prefill: the long job yields every 500 ms
t, waits2 = 0, {}
remaining = dict(jobs)
order = ["short-1", "long", "short-2", "short-3"]
started = {}
while remaining:
for name in order:
if name not in remaining:
continue
if name not in started:
started[name] = t
step = min(500, remaining[name])
remaining[name] -= step
t += step
if remaining[name] == 0:
del remaining[name]
print("chunked, prefill starts (ms):", started)
assert started["short-2"] == 700 and started["short-3"] == 900
queue_two_ways()
Without chunking, the second short request waits 4,200 ms before its prefill starts. With chunking, it starts at 700 ms. The long job still finishes. It just stops hogging the road. That is what chunked prefill buys, measured in milliseconds, not adjectives.
This one stayed with me because the model was fine, the GPUs were fine, and every user was still waiting.
After a product change, users could paste a whole document into the chat box and ask questions about it. One afternoon, time to first token went from 300 ms to over 4 seconds for everyone, and the queue kept growing. The model was healthy. The GPUs were healthy. The dashboard showed high utilization and no errors. Users saw a spinner that never turned into words. The app logs showed nothing at all, because the requests were still running, just very slowly, and slow requests do not log errors.
Three things. First, a maximum prompt length enforced at the gateway, with a clear error message, so a long prompt fails fast instead of blocking the line for everyone. Second, a per-request KV memory budget, so no single request can hold most of the GPU. Third, an alert on queue depth and on time to first token at p99, not just on average latency. The average hid this failure completely, because most requests were short and fast. The p99 showed it in the first minute. A serving dashboard that only watches averages is a dashboard that lies.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real serving setup. Take a minute with it before you read my approach.
Design the serving setup for an internal chat assistant used by 5,000 engineers. It answers questions about internal docs and code. The target is p95 time to first token under 500 ms with 200 engineers asking at the same time. You have a model that reads 512-token prompts in 180 ms of prefill, writes at 25 ms per token, and needs 0.5 MB of KV memory per prompt token. Sketch the serving setup and the budget.
Start with the budget from this lesson. A short question pays tokenize 5, queue 20, prefix cache check 10, prefill 72 with a 60 percent cache hit, and the first word appears at 107 ms. That is inside the 500 ms target with room to spare. Then size the workers with Little's Law from Lesson 35. Requests in the system equal the arrival rate times the time in the system. At 200 concurrent users and about 1.7 seconds per answer, that is 200 requests inside at once, so I plan the GPUs and the KV memory for 200 plus headroom, and I alert at 80 percent. KV memory is the binding number: 200 requests times 512 prompt tokens times 0.5 MB is about 51 GB of KV cache, plus the model weights on top. If one GPU holds 80 GB, the weights take roughly half and the KV cache takes the rest, so one GPU is not enough. I plan two GPUs per replica at minimum, three for headroom, and I cap prompts at 4,096 tokens so one long paste cannot eat the memory for everyone else. I turn the prefix cache on, because every request shares the same system instruction, and I set the batch cap at 16 so per-token latency stays near 25 ms instead of climbing toward 40.
Batching bigger buys throughput and costs per-token latency, so I cap the batch where the p95 time to first token starts to move. Quantization to INT8 buys speed and memory and costs a little quality, so I test it on the real internal questions before I ship it. Splitting prefill from decode buys isolation and costs a network hop for the KV notes, so I do it only if long prompts are common. Caching the docs buys speed and costs freshness, so I say on the page how old the docs are. And the target is p95, not the average. An average of 200 ms with a p95 of 900 ms fails this design.
Saved in this browser, so the Lessons index can show your progress across the course.