A large language model is a program that learned patterns from a huge amount of written text, and then uses those patterns to write new text one piece at a time. That is the whole trick. It does not look up answers in a book. It predicts what comes next, piece by piece, based on everything it read while it was trained. I use one every week now. A teammate pastes an error log into our support chat and asks what broke. The model reads the log, reads the question, and writes back a plain answer with the most likely cause and the next command to run. It is not magic and it is not a database. It is a prediction machine, and once you see it that way, the latency, the cost, and the failure modes all make sense. This lesson follows one question from the moment you send it to the moment the answer comes back, and counts what each part costs.
Every hop charges rent in milliseconds, but this trip is different from the ones in earlier lessons. The reading hops are priced by the prompt, and the writing hop is priced by the answer, and the two do not move together. The bar below is a worked example for one answer, not a measurement from a live system, but the shape is honest: on a short question, writing the answer is most of the trip, about 1,800 ms of the 2,395 ms total. On a long prompt, reading the prompt takes over, about 2,600 ms before the first word. Step through the lesson or switch the scenario and watch the bar change. The cold start is slow in a different place entirely: the wait at the front and the slow writing at the back.
Extra picture · starts simple, builds to the full picture
Starts from scratch with a user and one model box (scratch); adds tokenizer and embeddings, then layers and attention, then the KV cache, then safety check and ledger; ends at the multi-user service with queue, cache and cost ledger (Master: running it for a team, and when it breaks at 2 AM). I read the picture first. The words under each step are only labels.
where the moving unit is nowalready visitednot reached yetRequest · the way thereResponse · the way back
Extra picture · starts simple, builds to the full picture
Starts from scratch with a pile of text and one prediction (scratch: predict the next word); builds through the training run to weights; ends with the trained model inside the inference service (Master: pretraining vs fine-tuning trade-offs). I read the picture first. The words under each step are only labels.
where the moving unit is nowalready visitednot reached yetRequest · the way thereResponse · the way back
The budget above is one fixed trip. This box is yours. Change the prompt length, change the answer length, change the writing speed, and watch the bill move. Then ask the question I ask before every feature that calls a model: how much of this answer is reading, and how much is writing?
How many tokens are in the prompt. A short question is about 200. A pasted document is about 4,000. Every token has to be read before the first word of answer, so this number drives the reading cost.
How many tokens the answer runs to. A short answer is about 120. A long one is 500. The model writes one token at a time, so this number drives the writing cost.
How fast the machine writes. A quiet GPU writes about 60 tokens a second or more. A busy shared GPU drops toward 20. This is the throughput number from Lesson 02, applied to a model.
Hit means the start of the prompt was already read on an earlier turn, so only the new tokens are read now. The reading cost drops to a quarter. Miss means the whole prompt is read from scratch. The cache only helps when the start of the prompt stays the same from one turn to the next.
The reading formula is a straight-line teaching approximation. In real life the reading cost grows faster than the prompt, because every token looks at every other token, so doubling a long prompt more than doubles the reading work. Go deeper explains this properly.
Try this: (1) Move input tokens to 4,000 and watch the total jump from 2,590 ms to 10,190 ms. The answer did not change. Only the prompt did. (2) Now flip the KV cache to Hit. The total falls to 4,190 ms, because the reading cost drops to a quarter. Same prompt, same answer, far less reading. That is what the cache is for.
Three questions. No score is saved anywhere — this is just you against the material.
I read this topic four times here. First as a user, then as someone counting the bill, then as someone running it for a team, then as the person who gets paged when it breaks.
Definition: A large language model is a program that writes text one piece at a time. It learned patterns from a huge amount of written text. It does not look up answers. It predicts the next piece, over and over, until the answer is done.
Types:
Uses: I use the basic level when the job is small and the answer is checked by a human. A teammate pastes an error log into chat and asks what broke. I ask the model to explain it in plain words. I read the answer before I act on it.
Dimensions: At low load, one user and a few calls a day, this level costs almost nothing and breaks almost never. What breaks is the answer itself. The model can be wrong and sound sure. I notice when a human reads the answer and says that does not match the log. The cost is per-token money on a hosted service, or a slow laptop on a local one. What I give up is control over where the data goes and what the answer costs, because I am renting someone else's machine.
Definition: At this level I stop treating a call as magic and start treating it as two bills. Reading the prompt is the first bill. Writing the answer is the second. The two bills behave differently, and that difference decides what I do next.
Types:
Uses: I use this level when the bill first shows up. A support bot carries its full chat history into every call and pays for the whole history on every turn. I cap the history, summarize the old turns, and keep the fixed instructions at the front so the cache can reuse them. The cost dropped by two thirds on a bot I watched, with no change to the model.
Dimensions: As load grows, the reading bill grows faster than the prompt, because every token looks at every other token. Doubling a long prompt more than doubles the reading work. What breaks is the cache. If the start of the prompt keeps changing, the prefix never matches and the cached work is thrown away. I notice on two numbers: cache hit rate near zero, and GPU memory used by the cache. The cost is money on every call and memory on the GPU. What I give up is convenience. Long context feels easy and costs a lot, so I pay for it in prompt discipline instead.
Definition: At this level the model is one hop in a service, not a tool on my laptop. I route questions, pick models, and watch dashboards. The work is the system around the model.
Types:
Uses: I use this level when a team depends on the answers. A support bot answers real customers. An on-call helper reads real logs. I need to know the latency, the cost per answer, and the failure rate, and I need them on a dashboard, not in my head.
Dimensions: At 10x traffic the GPU becomes the bottleneck and the queue becomes the metric. I watch tokens per second per request, queue wait time, and cache hit rate. What breaks is the queue. Users wait, then they retry, then the queue doubles. I notice it on the queue depth graph before the users notice it in the chat. The cost is GPU hours and per-token money, plus the on-call time to keep it healthy. What I give up is simplicity. Every router, cache, and fallback is one more thing to monitor and one more thing that can fail.
A prompt that does not fit is not shortened by the model. It is refused, or the middle is cut and nobody is told. I count before I send. The rough count is one token per four characters. I add the system prompt, the logs, and the question, and I compare the total to the window. If the total is over, I drop the middle of the logs and keep the newest lines and the oldest lines, because the middle is usually heartbeat noise.
The drop must be visible. I log how many characters I kept and how many I dropped. A silent drop is a wrong answer with no warning. My rule is simple. If I drop more than half the logs, I stop and ask a smaller question instead of sending a cut-down one.
Definition: At this level I own the contract. The budget is a number with an alert on it. The fallback is tested, not hoped for. And when the provider changes the model under me overnight, I find out from my own evals, not from a customer.
Types:
Uses: I use this level on the night something breaks. (Illustrative scenario) The hosted model starts returning shorter, worse answers at 1 AM. My eval score dropped at midnight, before any user complained. I flip the route to the fallback model, cap the traffic, and page the team in the morning with numbers, not guesses.
Dimensions: At 100x the model stops being the product and the system around it becomes the product. What breaks is the thing I did not measure: a model update, a quota change, a region outage. I notice it because the evals run every night and the budget alert fires before the money does. The cost is the most expensive one in this lesson: my time, the eval set, and the discipline to keep both current. What I give up is the freedom to wing it. At this level nothing about the model is casual, and that is the point.
A hosted model can change under the same name. The provider ships a new revision, the answers shift, and my evals are the only thing that notices. I pin the model name and the revision in config, and the script refuses to run if the live revision does not match the pin. A mismatch is a deploy event, not a surprise.
When the pin changes, I run the eval set before I trust the new revision. The eval set is small and fixed, so the run takes a minute. If the score moves, I read the failures by hand. A model update that nobody evaluated is a prompt change nobody reviewed.
The lesson above is one question through nine hops. Production is the rest of it. Here is what I actually spend my time on once the tokens are counted and the budget is written down.
Attention is a scoring game. Each token asks every other token one question: how much do you matter to me. The model scores every pair, turns the scores into weights that add up to one, and then mixes the tokens together using those weights. That mixing is what 'understanding' looks like inside the model. The cost is the pairs. With 200 tokens there are 40,000 pairs to score. With 4,000 tokens there are 16 million pairs. Doubling the prompt quadruples the attention work, because every new token has to be scored against every token that is already there. That is why long context costs so much, and why the simulator's straight-line formula is only an approximation. In real life the curve bends up. The practical rule I use is simple: a prompt twice as long costs more than twice as much to read, so I trim the prompt before I pay for a bigger machine.
The KV cache is the model's short-term memory for work it already did. When the model reads a token, it computes two sets of numbers for it, called the key and the value. If the same token shows up again in the same conversation, the model can reuse those numbers instead of computing them again. That is the whole cache. It works best when the start of the prompt stays the same from one turn to the next, because the cache is matched by prefix: same beginning, same cached work. It breaks when the beginning keeps changing. If every turn rebuilds the prompt with the newest message first, or reorders the history, the prefix never matches and the cache never hits. The cache also costs memory. Every token kept in the cache holds its key and value numbers on the GPU, so a long conversation eats GPU memory just by sitting there. I watch two numbers: the cache hit rate and the GPU memory used by the cache. A hit rate near zero with a big cache means the prefix is churning. A cache that fills the GPU means old conversations are not being evicted.
Pretraining is the long, expensive part. The model reads a huge amount of text and learns how language works: grammar, facts, patterns, style. It takes weeks on many machines and it is not something a team does casually. Fine-tuning is the short, cheap part. You take a pretrained model and train it a little more on your own examples, so it learns your task: your tone, your format, your runbook style. I think of it as hiring someone who already speaks the language and then teaching them your team's way of answering. Fine-tuning needs far less data, a few hundred good examples beats ten thousand sloppy ones, and it has a real risk: train too hard on your examples and the model forgets general things it used to know. I fine-tune for format and tone, not for facts. Facts change, and a fact baked into the weights goes stale. Facts belong in the prompt, where I can update them.
Long context is expensive three ways, and people usually count only one. First, reading it costs more than proportionally, because of the attention pairs. Second, it costs memory, because the KV cache holds numbers for every token in the context. Third, it costs money on every single call, because you pay for the input tokens every time, even the ones you sent last turn too. A chat that carries its full history into every call pays for the whole history on every turn, and the bill grows with the conversation. The fixes are boring and they work: cap the history to the last few turns, summarize the older turns into a short paragraph, and keep the unchanging instructions at the front so the KV cache can reuse them. I have seen a support bot's cost drop by two thirds from those three changes alone, with no change to the model.
At 10x the traffic, the GPU becomes the bottleneck and the queue becomes the metric. I watch tokens per second per request, queue wait time, and cache hit rate, and I scale by adding GPU capacity or by splitting the load across machines. At 100x, the model stops being the product and the system around it becomes the product: routing short questions to a small fast model and hard questions to a big slow one, caching answers to common questions so the model never sees them, and streaming everywhere so a slow answer still feels alive. The budget stops being a target and becomes a contract, and every hop in this lesson owns its share of it.
I run a smaller model on my own laptop for my Kubernetes helper, through a local runner. Nothing leaves the machine. No log line, no config file, no cluster data. The model is smaller and slower than the big hosted ones, and it is wrong more often on hard questions, but it answers the common ones fine and it costs nothing per token. The trade is honest: local buys privacy and zero per-call cost, and it costs quality and speed. I use local for anything with real data in it and hosted for anything where the answer quality matters more than where the data goes.
This short snippet estimates the time and the token bill for one call from the prompt length, the answer length, and the speed, so you can price a call before you send it:
# Price one LLM call before you send it.
# All numbers are a teaching example, not a vendor price list.
input_tokens = 200 # tokens in the prompt
output_tokens = 120 # tokens in the answer
tokens_per_sec = 60 # how fast the machine writes
price_per_1k_in = 0.002 # cost per 1,000 input tokens
price_per_1k_out = 0.008 # cost per 1,000 output tokens
prefill_ms = input_tokens * 2 # reading the prompt
generation_ms = output_tokens / tokens_per_sec * 1000 # writing the answer
total_ms = prefill_ms + 60 + generation_ms + 130 # + first token + fixed hops
cost = (input_tokens / 1000 * price_per_1k_in
+ output_tokens / 1000 * price_per_1k_out)
print(f"total: {total_ms:.0f} ms, cost: ${cost:.4f}")
# Try input_tokens = 4000 and watch the total and the cost.
# The answer did not change. The prompt did.
Four inputs, two outputs. The time is the prefill plus the first token plus the writing plus the fixed hops, exactly the simulator's formula. The cost is the input tokens times the input price plus the output tokens times the output price, and the output price is usually higher, because writing costs more than reading. I run this before I design a feature that calls the model in a loop, because a loop that looks cheap per call gets expensive at a thousand calls a day. If the number surprises you, the prompt is too long or the answer is too long. Fix the prompt first. It is the cheaper fix.
The guard below does two jobs. It estimates the tokens before the call, and it refuses to send a prompt that does not fit. It also checks the pinned model revision against the live one, and it stops if they differ.
Run it with python3 check_the_budget.py. The log pile is 600,000 characters of filler, so the count lands over the window on purpose.
python3 check_the_budget.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. check_the_budget.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.
Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.
python3 --version python3 -u check_the_budget.py python3 -m json.tool --help
Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.
def check_the_budget():
CONTEXT_WINDOW = 128000
SYSTEM_PROMPT = "You are an SRE helper. Answer from the logs only."
def estimate_tokens(text):
return len(text) // 4
logs = "x" * 600000 # 150k tokens of log lines
question = "why did pod api-7 restart?"
total = estimate_tokens(SYSTEM_PROMPT) + estimate_tokens(logs) + estimate_tokens(question)
print(f"estimated tokens: {total} / window {CONTEXT_WINDOW}")
assert total > CONTEXT_WINDOW
keep = (CONTEXT_WINDOW - estimate_tokens(SYSTEM_PROMPT) - estimate_tokens(question) - 500) * 4
print(f"logs kept: {keep} chars of {len(logs)}; the middle is dropped, not the end")
assert keep < len(logs)
PINNED = {"model": "qwen2.5-14b", "revision": "a1b2c3d4"}
live = {"model": "qwen2.5-14b", "revision": "a1b2c3d4"}
assert live == PINNED, "model moved under you; eval again before trusting answers"
print("pinned model matches live model")
check_the_budget()
The estimate is 150,018 tokens against a 128,000 window, so the guard fires. It keeps 509,928 characters of logs and drops the rest, and it says so in plain text. The pinned model matches the live model, so that check passes. Change the revision string and run it again. The script must stop.
This one stayed with me because the model was fine, the code was fine, and the bill tripled anyway.
A support chatbot started answering slowly and repeating itself. Answers that used to take three seconds took twelve. The answers got worse too: the bot quoted old ticket notes that had nothing to do with the question. The model had not changed. The code had not changed. The prompt had. A deploy two weeks earlier had started attaching the full ticket history to every question, so the model could 'see the whole story.' By the third message in a conversation, the prompt was thousands of tokens of old tickets. The reading cost ballooned, the KV cache never hit because the history was rebuilt in a different order every turn, and the model spent most of its attention on stale text. The users saw a slow bot that quoted the wrong ticket. The finance team saw the bill first, because the cost tripled before anyone filed a latency ticket.
Three things. First, a hard cap on prompt size, enforced in code, not by convention: if the prompt is over 2,000 tokens, the history is summarized before the call is made, and the call never goes out over the cap. Second, the trace now alerts on input tokens per request, not just on latency. A prompt that grows quietly is a budget leak, and the alert catches it the week it starts, not the month the bill lands. Third, the prompt is built in a fixed order every time, instructions first, then the summary, then the recent turns, so the KV cache prefix matches and the cache actually hits. The lesson I took from it is simple: the prompt is a system, not a string. It has a size, a shape, and a cost, and it needs an owner.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real system that calls a model. Take a minute with it before you read my approach.
Design an internal support assistant for engineers. It answers questions from the team's runbooks and past incident notes. It must cite the runbook it used, answer 5,000 questions a day, answer in under 5 seconds at p95, and stay inside a fixed monthly token budget. Sketch the system: what the model sees, where the runbooks live, and how you keep the cost under control.
I would ask five things before I sized anything. How fresh must the answers be: can a runbook from last month answer, or does it need yesterday's incident notes? How long are the runbooks, and how many of them are there? What happens when the model does not know: does it say so, or does it guess? Is 5 seconds the target for the first word or for the full answer, because streaming changes the answer? And what is the budget in tokens per day, because the budget decides the model size before anything else does?
Start with the numbers from this lesson. Say a question is 100 tokens and the runbook excerpt attached to it is 800 tokens, so the prompt is about 900 tokens. The answer is about 150 tokens. Prefill is 900 times 2 ms, about 1,800 ms. Writing is 150 tokens at 60 a second, about 2,500 ms. Add the first token and the fixed hops, about 190 ms, and one answer is about 4,500 ms, inside the 5-second target with 500 ms of headroom. For the day: 5,000 questions times 900 input tokens is 4.5 million input tokens, and 5,000 times 150 output tokens is 750,000 output tokens. At the teaching prices in the snippet, that is about 9 dollars of input and 6 dollars of output a day, about 450 dollars a month. Little's Law for the sizing: if a request spends 4.5 seconds in the system and 5,000 arrive a day, the average load is tiny, under one request at a time, but the peak hour carries most of it, so I size the GPU for the peak hour, not the daily average, and I alert when queue wait passes one second.
Searching the runbooks and pasting the best match into the prompt buys fresh answers and costs input tokens on every call. Fine-tuning the model on the runbooks buys shorter prompts and costs a stale model every time a runbook changes, so I search for facts and fine-tune only for format and tone. A small fast model buys speed and cheap tokens and costs answer quality on hard questions, so I route easy questions to the small model and hard ones to the big one. Caching answers to common questions buys the cheapest answer of all, no model call at all, and costs a cache that can go stale, so I set a short TTL and invalidate on runbook changes. Streaming buys a fast first word and costs a client that can render tokens as they arrive. And the citation is non-negotiable: an answer without a runbook link is a guess, and a guess in an incident is worse than no answer.
Saved in this browser, so the Lessons index can show your progress across the course.