Platform Engineering · Lesson 58

vLLM & Serving Engines

A language model is a program that writes text one word at a time. A serving engine is the program that runs that model for many users at once, on a GPU, and hands the words back as they are written. Without one, every user would need their own copy of the model, and the GPU would sit idle between words. With one, hundreds of users share the same GPU, and the engine's job is to keep that sharing fair and fast. I run a small model on my own machine for my k8s-ai-sre project, and even there I feel the same problem. One slow prompt blocks the next one. This lesson follows one prompt through vLLM, the serving engine I reach for first, and shows where the time goes and what breaks when the GPU runs out of room.

Difficulty: Advanced · Know this first: Lesson 57, Lesson 48
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one request that gets a 25-token reply, not a measurement from a live system, but the shape is honest: in most slow answers I have chased, prefill and decode eat almost the whole budget, and the scheduler wait is the silent third cost. Step through the lesson or switch the scenario and watch the bar change. The long prompt nearly doubles the trip without the model doing anything different. The preempted request pays less in total than the long prompt but feels worse, because the user watches the answer freeze halfway.

Extra picture · starts simple, builds to the full picture

Inside vLLM

Starts from scratch with one prompt and one GPU; adds the scheduler and batch, then the KV blocks, then the dashboard; ends with multi-GPU tensor-parallel and pipeline-parallel layouts (Master: a fleet of engines). I read the picture first. The words under each step are only labels.

Moving: a promptStops: 8From: HTTP front doorTo: Dashboard and autoscaler
HTTP front doorstep 1 of 8step 1 of 8
Tokenizerstep 2 of 8step 2 of 8
Schedulercontinuous batchingcontinuous batching
PrefillGPUGPU
KV block managerPagedAttention blocksPagedAttention blocks
DecodeGPUGPU
Streamingstep 7 of 8step 7 of 8
Dashboard and autoscalerstep 8 of 8step 8 of 8
Architecture

    where the moving unit is nowalready visitednot reached yeta step where something stops or breaksRequest · the way thereResponse · the way backFailure · the packet stops here

    Extra picture · starts simple, builds to the full picture

    When the GPU runs out of room

    Starts from scratch with a short prompt's clean trip; adds chunked prefill, then the memory-full pause; ends with the recompute-vs-swap choice and its user-visible cost (Master: fairness under load). I read the picture first. The words under each step are only labels.

    Moving: a long promptStops: 7From: Long prompt arrivesTo: Notes restored, decode resumes
    Long prompt arrivesstep 1 of 7step 1 of 7
    Chunked prefill, piece 1step 2 of 7Chunked prefill, piece 1
    Decode step for another request slips instep 3 of 7step 3 of 7
    Chunked prefill, piece 2step 4 of 7Chunked prefill, piece 2
    Memory fullrequest pausedrequest paused
    Recompute branchrebuild the notes) vs swap branch (notes to CPU memoryrecompute vs swap
    Notes restored, decode resumesstep 7 of 7step 7 of 7
    Process

      where the moving unit is nowalready visitednot reached yeta step where something stops or breaksRequest · the way thereResponse · the way backFailure · the packet stops here

      Change the load and watch the answer bend

      The budget above is three fixed stories. This box is yours. Push the prompt longer, crowd the GPU with more requests, drain the free memory, and watch the time to first token and the total bend. Then ask the question I ask before every deploy: which half of the answer is slow, the start or the stream?

      How many requests are running at once. Up to 8, there is no queue wait. Above 8, each extra request adds 12 ms of waiting and 6 ms to the decode total.

      How long the prompt is. Prefill costs about 0.35 ms per token, so a 512-token prompt costs about 180 ms and a 4,096-token prompt costs about 1,434 ms.

      How much KV cache is still free. At 30% or more, no extra cost. Below 30%, each missing point adds 10 ms of recompute to decode.

      Time to first token220 msfront door + tokenizer + scheduler + prefill + KV blocks
      Total for a 25-token reply500 msfirst token plus the 25-token stream
      Decode speed100 tokens / sec25 tokens in 250 ms
      Queue wait0 ms0 ms up to 8 requests, then 12 ms per extra request

      Try this: (1) Set prompt length to 4096, leave the rest at defaults. Prefill is 4096 times 0.35, about 1434 ms. Time to first token is about 1474 ms and the total is about 1754 ms. Same engine, same model. Only the prompt changed. (2) Set concurrent requests to 64, leave the rest at defaults. Queue wait is 56 times 12, 672 ms. Decode is 250 plus 56 times 6, 586 ms. The total is about 1508 ms. The model did not get slower. It got crowded. (3) Set free KV cache to 15 percent, leave the rest at defaults. The missing 15 points below 30 add 150 ms of recompute to decode, so decode is 400 ms and the total is 650 ms. That is what a preemption feels like from the user's side.

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      I read this topic four times here. First as someone who needs one model to serve many users, then as someone turning the dials, then as someone splitting the work across GPUs, then as the person who runs the whole fleet.

      Basic — What a serving engine does

      Definition: A serving engine is the program that runs a model for many users at once on a GPU. Without one, every user would need their own copy of the model and the GPU would sit idle between words. With one, hundreds of users share the same GPU, and the engine's job is to keep that sharing fair and fast.

      Types:

      • vLLM, the engine I reach for first, open and widely used.
      • TGI, a small simple setup for one model behind one API.
      • TensorRT-LLM, the fastest on the right hardware, and the most demanding to deploy.
      • Triton, the generalist that serves many kinds of models, not just language models.

      Uses: I use the basic level when I have a model and more than one user. I run a small model on my own machine for my k8s-ai-sre project and even there one slow prompt blocks the next one. A serving engine is the fix for that, at any size.

      Dimensions: At one user the engine is overhead. At a hundred users it is the product. What breaks is fairness: one long prompt freezes everyone else. I notice it when time to first token spikes for the users who asked short questions. The cost is a program to learn and a GPU to feed. What I give up is the simplicity of calling the model directly, in exchange for sharing that actually works.

      Intermediate — The five dials

      Definition: At this level I stop accepting the defaults and start setting the knobs. Five lines in a launch command decide whether the server degrades slowly under load or falls over.

      Types:

      • Max model length, capping how long a prompt plus its answer can be.
      • Max sequences, capping how many requests run at once.
      • GPU memory utilization, setting how much of the GPU the cache may use.
      • Chunked prefill, cutting a long prompt into pieces so decode steps can slip between them.
      • Max batched tokens, capping how many prompt tokens one step may process.

      Uses: I use this level when the first real traffic arrives and the defaults start hurting. A pasted log file should not freeze a whole product, so I turn on chunked prefill and cap the batch.

      Dimensions: As load grows, every dial is a trade between one user's speed and everyone's fairness. What breaks is the cache when the caps are too loose, or the queue when they are too tight. I notice it on time to first token and on preemptions per minute. The cost is tuning time and a few bad afternoons of load testing. What I give up is raw speed for the biggest request, in exchange for a server that stays up.

      Advanced — More than one GPU

      Definition: At this level one GPU is not enough, so the work splits. The split has two shapes, and picking the wrong one costs more than the extra GPUs.

      Types:

      • Tensor parallel, splitting each layer across GPUs, fast, and needing a very fast link between them.
      • Pipeline parallel, splitting the layers down a line, less chatter, but a slow stage holds up the rest.
      • Preemption by recompute, throwing notes away and rebuilding them later.
      • Preemption by swap, moving notes to CPU memory and bringing them back.

      Uses: I use this level when the model is too big for one GPU, or one GPU is too slow. I pick tensor parallel inside one machine where the link is fast, and pipeline parallel across machines where it is not. I prove one GPU is too small before I pay for either.

      Dimensions: As load grows, the link between GPUs becomes the bottleneck, not the GPUs. What breaks is the slow stage in a pipeline, or the preempted request that has to re-read its whole prompt. I notice it on time per token and on the preemption count. The cost is money and complexity, and a deploy that is harder to roll back. What I give up is the easy single-GPU world, in exchange for running models that would not fit otherwise.

      The KV cache is a fixed number of blocks, and every running request holds some. When the blocks run out, new requests wait, and running requests can be preempted and recomputed. The number to watch is free blocks, not GPU use. GPU use can sit at 100 percent while the cache is fine, and the cache can be empty while the GPU is idle.

      I read the numbers before I guess. Free blocks under 10 percent with a waiting queue means scale out or shed load. Free blocks under 10 percent with an empty queue means a few long requests are holding the cache, and the fix is a shorter max length, not a new node. The same number means two different fixes, and the queue tells me which one.

      Master — A fleet of engines

      Definition: At this level the engine is a fleet, not a process. Traffic splits by use case. Each pool has its own budget, its own alerts, and its own rollout. The engine choice matters less than the numbers, and the numbers are the same three graphs on every engine.

      Types:

      • Split pools, short chat prompts on one pool, long document prompts on another.
      • Per-pool budgets for time to first token and time per token.
      • A staged rollout of engine versions, with a rollback that works.
      • An engine comparison done on my traffic, not on someone else's benchmark.

      Uses: I use this level when the serving layer is a service other teams depend on. I watch time to first token, time per token, and preemptions per minute, and I pick the engine whose graphs look best on my traffic.

      Dimensions: At 10x the KV cache breaks first, so I add GPU memory before I add GPUs, and I cap the running requests so a spike degrades slowly. At 100x one deployment is not enough, and long prompts cannot be allowed to starve short ones. What breaks is the pool with no alert, or the version with no rollback. I notice it on the three graphs, and the cost is a team, not a person. What I give up is the freedom to swap engines on a whim. At this level the engine is infrastructure, and infrastructure changes slowly, on purpose.

      A serving fleet is many engines behind one load balancer, and an update must not drop a request that is already decoding. I drain before I kill. The engine stops taking new requests, finishes the ones it holds, and only then exits. The drain has a timeout, because a request that runs for ten minutes is not a request to wait for. It is a request to move.

      I roll one replica at a time, and I watch the error rate after each one. If the new engine serves bad answers, I stop the roll with most of the fleet still on the old version. A roll that touches every replica at once is not a deploy. It is a gamble.

      Go deeper

      The lesson above is one prompt through nine hops. Production is the rest of it. Here is what I actually spend my time on once the budget is written down and the dashboard is up.

      Chunked prefill

      Prefill is one big GPU step, and a huge prompt can block every other request while it runs. Chunked prefill cuts a long prompt into smaller pieces and runs one piece per step, letting decode steps for other requests slip in between the pieces. The long prompt still takes the same total work, but it stops hogging the GPU. The trade is latency for fairness. The long prompt's first token arrives later, and everyone else's tokens keep flowing. I turn this on whenever long prompts share a GPU with chat traffic, because one pasted log file should not freeze a whole product.

      Scheduling policies

      The default policy is first come, first served. It is fair and it is simple, and it has one weakness: a long prompt that arrived first can hold the batch while short requests wait behind it. Some engines let short requests jump the queue, or give priority to requests that are already decoding over requests still waiting for prefill. That second rule matters most. A request that is halfway through its answer has already paid its prefill cost. Pausing it throws that work away. A new request has paid nothing yet, so making it wait is cheaper. The rule I follow is simple: protect the requests that are already running, and queue the new ones. When preemption cannot be avoided, the engine has two ways to pause a request. Recompute throws the notes away and rebuilds them later, which costs GPU time. Swap moves the notes to CPU memory and brings them back, which costs transfer time. Recompute is simpler. Swap is cheaper when the notes are large. Either way, the user feels the pause.

      Multi-GPU: tensor parallel and pipeline parallel

      One GPU holds one copy of the model. When the model is too big for one GPU, or one GPU is too slow, the work splits across several. Tensor parallel splits each layer across GPUs, so every GPU does part of every step. It is fast, and it needs a very fast link between the GPUs, because the GPUs talk to each other at every step. Pipeline parallel splits the layers instead: the first GPUs run the early layers, the later GPUs run the later layers, and activations flow down the line. It needs less chatter between GPUs, but a request has to travel the whole pipeline, and a slow stage holds up the rest. I pick tensor parallel inside one machine where the link is fast, and pipeline parallel across machines where it is not. Both cost money and complexity, so I prove one GPU is too small before I pay for either.

      Comparing serving engines

      vLLM, TGI, TensorRT-LLM, and Triton all run models for many users, and they differ in what they optimize for. vLLM is the one I reach for first: it is open, it runs most open models, and its block-based KV cache and continuous batching are the ideas this lesson teaches. TGI is a solid choice when the deployment is a single model behind a simple API and the team wants a small, well understood setup. TensorRT-LLM is the fastest on the right hardware, and it asks the most in return: the model has to be compiled for the specific GPU, and that compile step makes deploys slower and rollbacks harder. Triton is the generalist: it serves many kinds of models, not just language models, and it fits teams that already run mixed workloads on it. The engine matters less than the numbers. Time to first token, time per token, and preemptions per minute are the same three graphs on every engine, and I pick the engine whose graphs look best on my traffic, not the one with the best benchmark on someone else's traffic.

      What changes at 10x and 100x

      At 10x the traffic, the KV cache is the first thing that breaks. I add GPU memory before I add GPUs, because memory is the limit, and I cap the running requests so a spike degrades slowly instead of collapsing. I also set a maximum prompt length, because one huge prompt at 10x traffic is ten huge prompts. At 100x, one deployment is not enough. I split traffic by use case: short chat prompts on one pool, long document prompts on another, so the long ones cannot starve the short ones. The budget stops being a target and becomes a contract, and time to first token owns its own alert, separate from time per token.

      A minimal vLLM launch

      A minimal vLLM launch for one GPU, with the dials this lesson teaches:

      vllm serve my-model
        --max-model-len 8192
        --max-num-seqs 32
        --gpu-memory-utilization 0.90
        --enable-chunked-prefill
        --max-num-batched-tokens 4096

      Reading the launch line, flag by flag

      vllm serve my-model starts the server and loads the model named my-model. serve is the subcommand. It says run the OpenAI-compatible HTTP server, not a one-off job. my-model is the model name. It is a plain argument, not a flag. Leave it off and the server loads a tiny default model instead. That is not the model you meant to serve.

      --max-model-len 8192 caps the longest prompt plus answer, in tokens. Without it the length comes from the model config. That can be far longer than the box can hold. A longer window eats KV cache, so this cap is what keeps one long request from starving the rest.

      --max-num-seqs 32 caps how many requests run at once. The default is 256. This lesson sets 32 because a small box cannot hold 256 long requests at once. Drop the flag and the server allows 256. That looks fine until the cache fills and requests start getting preempted.

      --gpu-memory-utilization 0.90 says the server may use 90 percent of the GPU memory. The default is 0.9, so this line matches the default. It is written out so the number is a decision, not a surprise. Set it to 0.5 and the server leaves half the GPU alone. The KV cache shrinks with it, and fewer requests fit.

      --enable-chunked-prefill is a switch with no value. It turns on chunked prefill, so a long prompt is cut into pieces and shares each step with requests that are already decoding. Without it a long prompt can hold a whole step and the requests behind it wait. Newer vLLM versions switch this on by default, so writing it out pins the behaviour instead of trusting the version.

      --max-num-batched-tokens 4096 caps how many prompt tokens one step may process. Without it the engine picks its own number, and a big prefill can still hog a step. Smaller means fairer steps and slower prefill. Bigger means faster prefill and longer waits for everyone else.

      Three flags the launch line does not show, and you will meet them on the first real deploy. --port 8000 sets the port the API listens on. The default is 8000, so the line works without it. You only write it when the port is taken or the setup expects another one. --host 0.0.0.0 binds every network interface, which is the default. That means the API answers on the whole network, not just your own machine. For a local test I bind 127.0.0.1 instead, so only this machine can reach it. --dtype auto picks the weight precision from the model. The default is auto. I write --dtype bfloat16 when I want the choice pinned down instead of guessed. Try it.

      vllm serve my-model --port 8001 --host 127.0.0.1
      curl http://127.0.0.1:8001/v1/models

      curl fetches a URL and prints what comes back. It has no flags here. The first line starts the server on port 8001, reachable only from this machine. The second asks it what models it serves. If the server is not running, curl says connection refused. That tells you the server is down. It does not mean the address is wrong.

      max-model-len caps how long a prompt plus its answer can be, so one request cannot claim the whole cache. max-num-seqs caps how many requests run at once, which is the batch size dial from the simulator. gpu-memory-utilization sets how much of the GPU memory the KV cache may use; 0.90 leaves room for the model weights and the runtime. enable-chunked-prefill turns on the fairness trick from above. max-num-batched-tokens caps how many prompt tokens one step may process, so prefill never hogs a whole step. Those five lines are the difference between a server that degrades slowly under load and one that falls over.

      Read the cache numbers, then decide

      The snippet below reads a small set of serving numbers, the kind the metrics endpoint reports, and turns them into a decision. The rule is written down: free blocks under 10 percent, or more than 10 requests waiting, means scale.

      Run it with python3 read_the_cache_numbers.py. Then change the free blocks to 200 and run it again. The decision must flip to OK.

      Reading the run command

      python3 read_the_cache_numbers.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. read_the_cache_numbers.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.

      Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.

      python3 --version
      python3 -u read_the_cache_numbers.py
      python3 -m json.tool --help

      Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.

      def read_the_cache_numbers():
          metrics = {"num_requests_running": 42, "num_requests_waiting": 18,
                     "gpu_cache_usage_perc": 0.91, "num_free_gpu_blocks": 37,
                     "num_total_gpu_blocks": 512}
          free_frac = metrics["num_free_gpu_blocks"] / metrics["num_total_gpu_blocks"]
          print(f"free KV blocks: {free_frac:.1%}; waiting: {metrics['num_requests_waiting']}")
          assert round(free_frac, 3) == 0.072
          decision = "SCALE" if free_frac < 0.10 or metrics["num_requests_waiting"] > 10 else "OK"
          print("decision:", decision)
          assert decision == "SCALE"
      
      
      read_the_cache_numbers()
      

      Free KV blocks are 7.2 percent, and 18 requests are waiting, so the decision is SCALE. The numbers decide, not a feeling. When the free blocks rise and the queue clears, the same script says OK, and that is the script I would wire into the autoscaler.

      One time it broke

      This one stayed with me because the dashboard was green for the wrong reason. GPU use was high, which looked like success, and the users were watching their answers freeze mid-sentence.

      What broke

      A chat feature launched to all users on a Tuesday morning. Traffic tripled in an hour. The serving pods looked busy and healthy, GPU use sat near 100 percent, and no pod crashed. But users saw their answers stop halfway, then resume after a few seconds, then stop again. Support tickets said the model was broken. The model was fine. The GPU memory was full, so the engine kept pausing requests halfway through their answers to make room for new ones, and each paused request had to redo its prefill before it could continue. The pauses fed on themselves. Every redo took GPU time, which made room scarcer, which caused more pauses.

      Timeline

      • 09:05Feature flag turns on for all users. Traffic climbs from 15 requests a second to 45.
      • 09:12First ticket: "the answer stops halfway." I check the pods. All healthy, GPU use near 100 percent.
      • 09:20I check time per token. It doubled. Then I check the preemption count. It went from zero to hundreds a minute. That is the clue.
      • 09:28I check KV cache free. It is near zero and never recovers between spikes. The GPU is full.
      • 09:35I lower the cap on running requests from 64 to 32. Preemptions fall, but the queue grows and time to first token climbs.
      • 09:50I add two more GPU pods to the deployment. The queue drains and preemptions stop.
      • 10:15I confirm from three accounts that answers stream without freezing, and the preemption count is back to zero.

      What changed after

      Three things. First, the cap on running requests is now set from the KV cache size, not from a round number. I worked out how many average requests fit in the cache and set the cap below that, with headroom. Second, preemptions per minute got its own alert, paging the on-call above a small threshold. Zero to anything is the earliest warning the GPU is full. Third, the launch checklist now includes a load test at three times the expected traffic, with long prompts in the mix. The old load test used short prompts only, so it never saw the memory pressure that real users brought.

      If you remember nothing else

      Design exercise

      This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real serving setup. Take a minute with it before you read my approach.

      The prompt

      Design the serving setup for an internal support chatbot. It must answer 50 requests a second, with p99 time to first token under 1 second and 25-token replies on average. Prompts average 800 tokens. You have a GPU pool, a load balancer, and the vLLM dials from this lesson. Sketch the setup and the budget.

      Clarifying questions to ask first

      • How long are the longest prompts, not just the average? A few 8,000-token prompts change the memory math completely.
      • Is 50 requests a second the steady load or the Monday morning peak?
      • What happens when the GPU pool is full: do new requests wait, get a busy message, or spill to a slower fallback?
      • How fresh must the answers be, and can any of them be cached? Repeated support questions are the easiest win in serving.
      • What is the failure we design for: one GPU pod down, or the whole pool slow?

      Rough sizing, worked

      Start with the budget from this lesson. An 800-token prompt costs about 280 ms of prefill (800 times 0.35). Add the front door, tokenizer, scheduler, and KV blocks, about 40 ms, and time to first token is about 320 ms, inside the 1 second target with room to spare. Decode for 25 tokens at 10 ms each is 250 ms, so a request lives in the system about 570 ms end to end. Little's Law says the requests inside at once equal the arrival rate times the time in the system, so 50 times 0.57 is about 29 requests inside at once. One GPU running 32 sequences holds that, but with no headroom, so I plan two GPU pods and alert at 80 percent of the running-request cap. Memory: an 800-token prompt plus a 25-token answer needs about 825 tokens of KV blocks per request, so 32 running requests need room for about 26,400 tokens of notes. I check that against the GPU's KV cache size before I sign off, and I set the maximum prompt length so one huge prompt cannot eat the pool.

      Trade-offs to name

      A bigger running-request cap buys throughput and costs time per token, so I cap it from the cache size. Tensor parallel across two GPUs buys speed for one request and costs a fast link and a harder deploy, so I start with two separate replicas instead. Caching repeated support answers buys speed and costs freshness, so I cache only the stable help articles, never live account data. And the target is p99, not the average. An average time to first token of 300 ms with a p99 of 3 seconds fails this design.

      Saved in this browser, so the Lessons index can show your progress across the course.

      Next in this series
      Lesson 59 — Temperature & Decoding

      You know how the engine runs the model. Next you learn how the model picks its next word, and why the same prompt can give a different answer every time.