A language model does not normally produce a single inevitable next token. Given a prefix, it assigns scores to many possible tokens. A decoding algorithm then decides how to turn those scores into the next output.
That last step matters. If you sample too freely, a model can drift into unlikely continuations. If you restrict sampling too aggressively, outputs can become repetitive or lose useful variation. Parameters such as temperature, top-k, and top-p control different parts of this trade-off, so treating them as interchangeable “creativity settings” leads to confusing results.
This article builds a practical mental model for these controls. You will learn what each one changes, how they interact, why the same setting behaves differently across prompts, and how to tune sampling without assuming that randomness improves model knowledge or correctness.
Start with the next-token distribution
Suppose a model sees:
The service returned an HTTPFor a simplified example, imagine its next-token probabilities are:
500 0.45
404 0.25
200 0.15
503 0.10
302 0.05These numbers are only a teaching example. Real language models usually choose among vocabularies containing many more tokens, and tokens do not necessarily correspond to complete words or numbers.
A greedy decoder chooses the highest-probability token, so it would select 500 every time in this example. A sampling decoder instead draws a token from a probability distribution. 500 remains the most likely result, but other tokens can sometimes be selected.
The important point is that temperature, top-k, and top-p do not add new knowledge to the model. They transform or restrict the distribution from which the decoder selects.
Temperature changes relative probability
Before probabilities are produced, a language model emits numerical scores called logits. Softmax converts those logits into probabilities. Temperature modifies the logits before that conversion:
p_i = exp(z_i / T) / sum_j exp(z_j / T)where z_i is the logit for token i and T is a positive temperature.
When T < 1, dividing by temperature increases the gaps between logits. After softmax, high-scoring tokens receive more probability and low-scoring tokens receive less. The distribution becomes sharper.
When T > 1, the logit gaps shrink. The resulting distribution becomes flatter, giving lower-ranked tokens more chance to be sampled.
At T = 1, temperature leaves the logits unchanged.
For example, consider three logits:
A: 2
B: 1
C: 0At temperature 1, their softmax probabilities are approximately:
A: 0.665
B: 0.245
C: 0.090At temperature 0.5, the effective logits become 4, 2, 0, producing approximately:
A: 0.867
B: 0.117
C: 0.016The ranking did not change. The decoder simply became much more likely to choose the already preferred token.
Temperature is not a correctness dial
Lower temperature can make output more stable when the model already places high probability on a good continuation. It does not prove that the preferred continuation is correct. If the model confidently favors a factual error, sharpening the distribution can make that error more repeatable.
Likewise, raising temperature does not make a model reason more creatively in any guaranteed sense. It increases the relative probability of lower-scoring alternatives. Some alternatives may be useful; others may be incoherent or wrong.
Top-k keeps a fixed number of candidates
Top-k sampling sorts candidate tokens by probability and keeps only the k highest-ranked candidates. The remaining probabilities are set to zero, and the retained probabilities are renormalized before sampling.
Using the earlier distribution:
500 0.45
404 0.25
200 0.15
503 0.10
302 0.05with k = 3, only these tokens remain:
500 0.45
404 0.25
200 0.15Their original probability mass sums to 0.85. After renormalization, the sampling probabilities become approximately:
500 0.529
404 0.294
200 0.176Top-k therefore limits the candidate set rather than directly changing the relative scores among retained tokens.
A small k can prevent a long tail of unlikely tokens from being sampled. The limitation is that a fixed candidate count ignores how concentrated the distribution is. Sometimes the model has one overwhelmingly likely continuation; elsewhere dozens of alternatives may all be plausible. The same k applies to both situations.
Top-p keeps enough candidates to cover probability mass
Top-p sampling, also called nucleus sampling, adapts the candidate count to the shape of the distribution.
The decoder sorts tokens from most to least probable, then keeps the smallest leading set whose cumulative probability reaches at least p. It renormalizes that set and samples from it.
With the same example and p = 0.8:
500 0.45 cumulative 0.45
404 0.25 cumulative 0.70
200 0.15 cumulative 0.85The first two candidates are not enough because they cover only 0.70 probability mass. Adding 200 raises the cumulative mass to 0.85, so the nucleus contains three tokens.
Now consider a much more concentrated distribution:
500 0.92
404 0.03
200 0.02
503 0.02
302 0.01With p = 0.8, the first token alone already exceeds the threshold. The nucleus can therefore contain only 500.
This adaptive behavior is the central difference from top-k. Top-k fixes the number of candidates. Top-p fixes the amount of original probability mass the candidate set must cover.
The boundary token is included
A common implementation mistake is to remove the token that makes cumulative probability cross the threshold. In the first p = 0.8 example, that would incorrectly keep only 500 and 404, whose cumulative probability is 0.70.
The usual nucleus definition keeps the smallest prefix whose cumulative mass is at least the threshold. Exact API behavior should still be checked because serving libraries can expose additional filtering rules or special parameter semantics.
The controls can be combined
A decoding pipeline can apply temperature and truncation together. Conceptually, a common sequence is:
model logits
-> divide logits by temperature
-> convert to probabilities
-> restrict candidates with top-k and/or top-p
-> renormalize
-> sampleThis ordering explains an important interaction: temperature can change which tokens fall inside a top-p nucleus because it changes their probabilities before cumulative mass is calculated.
Suppose a high temperature flattens a distribution. More tokens may then be required to reach p = 0.9. A low temperature may concentrate most probability on a few tokens, shrinking the nucleus.
Top-k and top-p can also be used together. In that case, the effective candidate set is restricted by both rules. The exact order and parameter semantics are implementation details, so applications should follow the documentation of the inference system they actually use rather than assuming every API applies filters identically.
Separate deterministic decoding from low-randomness sampling
Developers often describe a low temperature as “deterministic,” but these are different ideas.
If an implementation still samples from a distribution containing multiple tokens, repeated runs can produce different outputs even when temperature is low. Lower temperature merely concentrates probability more heavily on high-scoring tokens.
True deterministic behavior depends on the decoding algorithm and serving implementation. Greedy decoding selects the highest-scoring candidate rather than drawing randomly. Some APIs expose a special temperature value or a separate option that maps to deterministic decoding, while others define different constraints. Do not assume that a particular numeric value has universal API semantics.
Even deterministic token selection does not necessarily imply bit-for-bit reproducibility across every environment. Model revisions, numerical kernels, hardware, batching, and serving changes can affect results unless the provider explicitly guarantees reproducibility under stated conditions.
Tune sampling around the application’s failure cost
There is no universal temperature, k, or p that is correct for every task. A useful tuning process starts with what kinds of variation the application can tolerate.
For structured extraction, classification-like prompts, or code transformations with narrow expected outputs, unnecessary sampling variation often makes evaluation and debugging harder. Greedy or tightly constrained decoding may be a better baseline. If the output must obey a schema, grammar-constrained or structured decoding can address validity more directly than merely lowering temperature.
For brainstorming, alternative phrasings, or generating multiple candidates for later ranking, some sampling diversity can be useful. The application can deliberately generate several candidates and evaluate them rather than expecting one random sample to be both diverse and reliable.
For factual question answering, changing sampling settings is not a substitute for grounding, retrieval, tool use, or verification. Sampling controls choose among continuations represented by the model’s distribution; they do not provide evidence that a statement is true.
A practical experiment might look like this:
- Build a representative evaluation set before changing decoding parameters.
- Establish a simple baseline, such as greedy decoding or the serving system’s documented default.
- Change one control at a time so its effect is observable.
- Measure task quality as well as output diversity, latency, and failure rate where relevant.
- Test repeated samples for prompts where randomness is enabled.
The important measurement depends on the application. Exact-match accuracy may matter for extraction, while candidate diversity and downstream acceptance rate may matter for ideation. A single generic “creativity” score rarely captures these different goals.
Watch for common sampling mistakes
Changing several controls at once
If temperature, top-k, and top-p all change between experiments, it becomes difficult to identify which change caused an improvement or regression. Start from a known baseline and vary one dimension unless you are deliberately running a joint parameter search.
Using a high temperature to fix repetitive output
Repetition can come from the model, prompt, context, stopping rules, or decoding configuration. Higher temperature may reduce some repeated patterns by spreading probability mass, but it can also increase unrelated errors. Diagnose the failure before treating randomness as the remedy.
Using a low temperature to prevent hallucinations
A lower temperature can make generations more consistent, but consistency and factuality are different properties. Confidently wrong continuations can remain confidently wrong. Evaluate factual failures directly and add grounding or verification when the application requires it.
Comparing settings with one generation per prompt
Sampling is stochastic. A single output can make a setting look unusually good or bad by chance. When comparing stochastic configurations, run enough samples to characterize the variation that matters for your application.
Assuming parameter names imply identical behavior
Inference APIs differ. Some allow temperature with top-p, some expose top-k, some use provider-specific defaults, and some implement deterministic modes separately. Treat the mathematical concepts in this article as a mental model, then verify the actual API contract before relying on edge-case behavior.
When simpler decoding is enough
Sampling controls are useful when you genuinely need to trade concentration for diversity. They are less useful when the output space is already narrow and correctness can be specified directly.
If a task has one expected label, choose among allowed labels rather than adding randomness. If a response must satisfy a machine-readable schema, constrained generation or validation with retry logic may be more appropriate. If you need several distinct ideas, explicit multi-candidate generation can be easier to evaluate than hoping a single high-temperature sample is useful.
The simplest decoding strategy that meets the product requirement is usually the easiest to test and operate. Add sampling freedom because the application benefits from variation, not because language models are assumed to require randomness.
Conclusion
Temperature, top-k, and top-p control different stages of token sampling. Temperature reshapes relative probabilities, top-k limits the candidate count, and top-p adapts the candidate set to cumulative probability mass. Their effects depend on the model’s distribution at each generation step, which is why the same numeric settings can behave differently across prompts.
Use these controls as decoding tools rather than quality guarantees. Start from a measurable baseline, introduce randomness only where variation has value, and evaluate the failures that matter to the application. That mental model makes sampling settings easier to reason about than treating them as a single creativity knob.