Here is the whole lesson in one sentence. A language model does not pick the next word. It ranks every word it knows, and then a small set of settings decides how that ranking turns into a choice. Those settings are called decoding, and temperature is the most famous knob in the set. Turn it down and the model gives you the same safe answer every time. Turn it up and the same prompt gives you a different answer on every run, sometimes a good one, sometimes a strange one. I learned this the way I learn most things in this job. A tool I ran on my own machine gave me a different answer to the same question twice, and I assumed the tool was broken. It was not broken. I had left temperature at 1.2. This lesson walks one prompt through the whole decoding trip, knob by knob, so you know exactly which setting to reach for when the output is too boring, too wild, or not the same twice.
Every hop charges rent in milliseconds. The bar below is a worked example for one answer, not a measurement from a live system, but the shape is honest: the decoding knobs cost about 1 ms each, and the token loop costs almost everything. Step through the lesson or switch the scenario and watch the bar change. The wild scenario nearly doubles the trip without the model doing anything different. Same model, same prompt, same ranking. Only the knobs changed, and the bill doubled.
Extra picture · starts simple, builds to the full picture
Starts from scratch with one ranked word list and one draw (the model ranks, a setting picks); adds one knob per step; ends with schema-constrained decoding where the format is guaranteed (Master: decoding as a contract). I read the picture first. The words under each step are only labels.
where the moving unit is nowalready visitednot reached yeta step where something stops or breaksRequest · the way thereResponse · the way backFailure · the packet stops here
The budget bar above shows three fixed stories. This simulator lets you write your own. It shows one real decision the model makes: after the words "The capital of France is", which word comes next? Five candidate words, with the scores the model gave them. Move a slider and watch the probabilities move, the pool shrink or grow, and the verdict change. The numbers are a teaching example, not live measurements. The shape is the lesson: temperature spreads the choice, top-k and top-p cut the list, and the draw does the rest.
The capital of France is ___
Scores the model gave each candidate word: Paris 6.0 · the 5.5 · a 4.5 · Lyon 3.5 · banana 1.0
Divides each score by the temperature, then turns the results into probabilities. Lower temperature makes the gaps bigger, so the top word pulls ahead. Higher temperature makes the gaps smaller, so the words bunch up. At temperature 0 the top word gets 100 percent and the rest get 0.
Sorts the words from most likely to least likely, walks down adding probabilities, and stops when the total reaches top-p. Only the words walked past stay in the pool. The kept words are then re-scaled so their probabilities add up to 100 percent.
Keeps only the k highest-scored words and throws the rest away before top-p runs. With 5 candidate words and top-k at 40, nothing is cut. With top-k at 2, only Paris and the survive, whatever top-p says.
When on, words already used in the answer have their scores lowered before the pick, so loops are less likely and the answer finishes sooner. When off, the model can repeat itself, so the estimated answer length grows.
Try this: (1) Set temperature to 0. The pool collapses to Paris at 100 percent and effective choices reads 1.0. That is greedy mode. (2) Set temperature to 2.0 and top-p to 1.0. Paris falls to about 38 percent, banana climbs to about 3 percent, and effective choices rises to about 3.5. Same scores, same model. Only the knobs changed. (3) Set top-k to 1 at any temperature. The pool is Paris alone, and temperature stops mattering. That is the guardrail doing its job.
Three questions. No score is saved anywhere — this is just you against the material.
I read this topic four times here. First as one knob on one model call, then as the full set of knobs, then as a guarantee on the format, then as a contract across the whole service.
Definition: A language model does not pick the next word. It ranks every word it knows, and then a small set of settings decides how that ranking turns into a choice. Temperature is the most famous knob in that set. Turn it down and the model gives you the same safe answer every time. Turn it up and the same prompt gives you a different answer on every run.
Types:
Uses: I use the basic level every time I run a local model. (Illustrative scenario) A tool on my own machine gave me a different answer to the same question twice, and I assumed the tool was broken. It was not broken. I had left temperature at 1.2. Now I set temperature 0 for anything I need to repeat, and I save the higher settings for drafts a human will edit.
Dimensions: At low load the knob is free and the only cost is a surprising answer. What breaks is repeatability. A test that passes on Monday and fails on Wednesday is worse than no test, and a high temperature is usually why. I notice it when the same input gives a different output twice. The cost is nothing in money and everything in trust, because a non-deterministic tool is a tool I cannot debug. What I give up with temperature 0 is variety. There is none, and for a test that is exactly right.
Definition: Temperature is one knob. The full set is small, and each knob does one job. Once I know them, the whole behavior of a model call is visible in one request body, in version control, not hidden in a dashboard slider.
Types:
Uses: I use this level when I write a request body I will run more than once. A status update for a backup job gets temperature 0, a fixed seed, and a 60-token cap. The same body works for a draft when I raise the temperature, because the guards are already there.
Dimensions: As traffic grows, the knobs become a cost decision. A high temperature means longer answers, because the model rambles and misses the stop token, and longer answers mean more tokens and more GPU time. What breaks is the runaway answer: no stop token, no cap, and the model writes until the budget does. I notice it on tokens per answer climbing. The cost is small per call and large in aggregate. What I give up is the convenience of defaults. Defaults are set for a demo, not for my service.
Definition: At this level I stop hoping the model writes valid JSON and start building the format into the pick itself. Constrained decoding removes every token that would break the format before the draw happens. The model cannot produce a trailing comma, because the comma is not in the pool at that position.
Types:
Uses: I use this level for anything a machine reads next: API responses, function calls, database writes. A model that writes free text is a writer. A model that writes JSON under a schema is a form filler, and I treat it like one.
Dimensions: As load grows, constrained decoding adds a small cost per step and removes a large failure mode. What breaks is sense, not shape. The grammar guarantees the shape. It does not guarantee the answer is right, and a valid answer that is wrong is still wrong. I notice it when the parser stops failing and the wrong answers start passing, which is why I still eval the content. The cost is the schema work and a little latency. What I give up is the model's freedom to phrase things its own way, and for machine-read output that freedom was never worth having.
A seed does not make sampling deterministic across machines. The same seed on a different GPU, or with a different batch size, can give a different answer, because the floating point math runs in a different order. I test reproducibility on the same machine, with the same batch size, and I record the machine in the test.
For the cases where the answer must be the same every time, I do not rely on a seed. I use greedy decoding at temperature zero and I compare the full output string, not a summary. If the string changes, something in the stack changed, and I find out what before I ship.
Definition: At this level decoding settings are a contract, not a preference. Support tools, evals, and anything with a retry must give the same answer for the same input, or the retry logic and the debugging both break. So I pin temperature to 0 and fix the seed for those paths, and I keep the creative settings behind a separate route with its own budget.
Types:
Uses: I use this level when a service has two kinds of traffic and one set of knobs cannot serve both. One service, two decoding profiles, and the eval suite runs cold so a pass means something.
Dimensions: At 10x the traffic, decoding settings are a cost line. I watch tokens per answer the way I watch latency, because they are the same bill. At 100x, determinism becomes a contract, and a change to a decoding default is a deploy with an eval gate. What breaks is the silent drift: someone raises a temperature in a dashboard, the evals start flaking, and nobody connects the two. I notice it because the config is in version control and the evals run on a schedule. The cost is the discipline to keep two profiles and to never let a slider hide in a dashboard. What I give up is the freedom to tweak a knob on a whim. At this level a knob is a production change, and it gets treated like one.
One decoding config for every request is a mistake. A classification needs temperature zero and a short cap. A draft needs temperature 0.7 and room to write. I keep one config file per route, in version control, with a name on each.
A config change is a deploy. I roll it to a slice of traffic first and I compare the eval score and the parse rate against the old config. If either falls, the config rolls back with one revert. A config edited in a dashboard has no revert, and no revert means no rollback.
The lesson above is one prompt through nine hops. Production is the rest of it. Here is what I actually spend my time on once the knobs are understood and the budget is written down.
Logit bias is a thumb on the scale. It adds a fixed number to the score of a word you choose, before temperature and the cutoffs run. A positive bias pushes a word up the ranking. A negative bias pushes it down. A bias of minus 100 bans the word completely. I use it for small, concrete jobs. Ban the word that keeps showing up in a template. Push the word yes up when the answer must be yes or no. It is a scalpel, not a strategy. If I need a bias list with fifty entries, the prompt is wrong, not the decoding.
Structured output is the grown-up version of hoping the model writes valid JSON. Constrained decoding builds the format into the pick itself. At every step, the decoder looks at the schema and removes every token that would break it. If the schema says the next character must be a digit, every letter is removed from the pool before the draw. The model cannot produce a trailing comma, because the comma is not in the pool at that position. I use this for anything a machine reads next: API responses, function calls, database writes. The cost is small and the failure mode it removes is large. A model that writes free text is a writer. A model that writes JSON under a schema is a form filler, and I treat it like one.
Grammar-constrained generation is the same idea with a grammar instead of a JSON schema. You give the decoder a small grammar, a set of rules for what a valid answer looks like, and it only ever picks tokens that keep the answer inside the rules. I use it when the output has a shape JSON cannot describe cleanly, like a query language or a config file. The warning is the same as with any constraint. The model can still pick a valid answer that is wrong. The grammar guarantees shape. It does not guarantee sense.
Sampling hurts in exactly three places, and I have been bitten in all three. Evals, because a sampled eval measures luck, not quality, and a suite that passes on Monday and fails on Wednesday is worse than no suite at all. Code, because code has one right answer and a creative alternative is a bug with good spelling. And JSON or any machine-read format, because one wrong token breaks the parser and the whole response is garbage. In all three places the fix is the same: temperature 0, a fixed seed, and constrained decoding where the format matters. Save the sampling for the places where variety is the product: drafts, brainstorms, and first versions a human will edit.
At 10x the traffic, decoding settings become a cost decision. A high temperature means longer answers, because the model rambles and misses the stop token, and longer answers mean more tokens, and more tokens mean more GPU time and more money. I watch tokens per answer the way I watch latency, because they are the same bill. At 100x, determinism becomes a contract. Support tools, evals, and anything with a retry must give the same answer for the same input, or the retry logic and the debugging both break. I pin temperature to 0 and fix the seed for those paths, and I keep the creative settings behind a separate route with its own budget. One service, two decoding profiles, chosen by the caller, not by accident.
This is a request to a local model, the way I run it for my own tooling:
{
"model": "qwen2.5:7b",
"prompt": "Write a one-line status update for the nightly backup job.",
"stream": false,
"options": {
"temperature": 0.0,
"top_p": 0.9,
"top_k": 40,
"repeat_penalty": 1.1,
"seed": 42,
"num_predict": 60,
"stop": ["\n\n"]
}
}
Temperature 0.0 means the top word always wins, so the answer is repeatable. Top-p 0.9 and top-k 40 are the guardrails, and they do nothing at temperature 0, but they are there so the same request body works when I raise the temperature for a draft. Repeat penalty 1.1 keeps the answer moving. Seed 42 fixes the draw, so a rerun gives the same answer. Num predict 60 caps the answer at 60 tokens, so a missing stop token cannot run away. Stop is a list of strings that end the answer early. Six settings, each doing one job, and the whole behavior is visible in one request body. That is how I want decoding configured: in the open, in version control, not hidden in a dashboard slider.
Temperature does one thing: it reshapes the odds before the pick. The snippet takes four words with fixed odds and shows what three temperatures do to them. Then it draws five samples at temperature 1.0, with a fixed seed, so you get the same five words I got.
Run it with python3 same_words_three_temperatures.py.
python3 same_words_three_temperatures.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. same_words_three_temperatures.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.
Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.
python3 --version python3 -u same_words_three_temperatures.py python3 -m json.tool --help
Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.
def same_words_three_temperatures():
import random
vocab = ["restart", "memory", "config", "network"]
probs = [0.50, 0.30, 0.15, 0.05]
def temper(p, t):
scaled = [x ** (1.0 / t) for x in p]
s = sum(scaled)
return [round(x / s, 3) for x in scaled]
for t in (0.2, 1.0, 1.5):
print(f"T={t}: {temper(probs, t)}")
assert temper(probs, 0.2) == [0.926, 0.072, 0.002, 0.0]
assert temper(probs, 1.0) == probs
random.seed(3)
picks = [random.choices(vocab, weights=temper(probs, 1.0))[0] for _ in range(5)]
print("T=1.0 samples:", picks)
assert picks == ["restart", "memory", "restart", "memory", "memory"]
same_words_three_temperatures()
At T=0.2 the top word takes 0.926 of the odds, so the answer is nearly fixed. At T=1.0 the odds are the original [0.5, 0.3, 0.15, 0.05]. At T=1.5 the odds flatten toward even. The five samples are restart, memory, restart, memory, memory. Same seed, same machine, same five words. Change the seed and they change.
This one stayed with me because nothing was broken. The model was fine, the prompt was fine, and the answers were still wrong half the time.
A support team used a small drafting tool that turned a ticket into a suggested reply. The tool worked well for weeks. Then the team lead noticed the same ticket got a different draft every time an agent hit retry. Sometimes the draft was good. Sometimes it offered a refund the policy did not allow. Sometimes it quoted a step that did not exist. The agents stopped trusting it, and they were right not to trust it. The tool was not broken. It was rolling dice on every retry and nobody had noticed, because the first few rolls had been lucky.
Three things. First, two decoding profiles in the config, in version control, not in a dashboard. One is deterministic, temperature 0 with a fixed seed, for anything a customer might see or an eval might check. The other is creative, temperature 0.7, for drafts a human will edit. The caller picks the profile, and the default is deterministic. Second, the golden set runs on every deploy, and it compares the output exactly. A temperature change is a behavior change, and it goes through review like any other behavior change. Third, the tool logs its decoding settings with every request, next to the model name and the prompt version. When an answer looks wrong, the first thing I read is the settings that produced it, not the model. The model was never the problem here. The settings were, and they were invisible until I made them visible.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real decoding config. Take a minute with it before you read my approach.
Design the decoding settings for two features in one product. Feature one is a code-completion service inside an editor: it must return valid code, fast, and the same completion for the same cursor position. Feature two is a brainstorming tool: it must give five different opening lines for a status update, and the user picks one. You have one model, one API, and a latency budget of 300 ms at p99 for a 20-token answer. Sketch the settings for each feature, and say what you would measure.
Start with the budget from this lesson. The fixed hops are about 16 ms: tokenize 2, score 8, and 1 ms each for temperature, top-k, top-p, pick, and penalty, with 2 ms for the penalty check. That leaves 284 ms for the loop inside a 300 ms budget. At about 19 ms per token, that buys about 14 tokens in the loop, so a 20-token answer at about 396 ms is already over budget, and I say so honestly. To hit 300 ms I need about 15 tokens, so I cap the code completion at 15 tokens and stream the first token early, because the user sees the first token in about 35 ms and reads while the rest arrives. For the brainstorming tool I run five short generations of 8 tokens each, in parallel, at temperature 0.9, which is 5 times about 168 ms of work but overlapped, so the wall clock stays near 200 ms. Little's Law for the sizing: 100 completions a second at 0.3 seconds in the system is 30 requests inside at once, so I plan the model workers for 30 concurrent, plus headroom, and I alert at 80 percent.
Temperature 0 buys repeatability and costs variety, which is the right trade for code and the wrong trade for brainstorming. Constrained decoding buys valid output and costs flexibility, because the model can only pick tokens the schema allows, and a schema that is too tight strangles good answers. A fixed seed buys reproducibility and costs nothing, so I set one everywhere and only remove it where I want variety on purpose. Streaming buys perceived speed and costs a harder cancellation story, because a user who cancels mid-stream still paid for the tokens generated so far. And the target is p99, not the average. An average of 200 ms with a p99 of 900 ms fails this design, and the p99 is where the runaway answers live.
Saved in this browser, so the Lessons index can show your progress across the course.