A language model can produce very different continuations from the same prompt even when its weights and context haven’t changed. One of the controls behind that variation is temperature.

Temperature is often described as a creativity knob. That description is convenient but incomplete. Temperature doesn’t add ideas to a model, improve its knowledge, or directly control factual accuracy. It changes the probability distribution used to choose the next token. The practical effect depends on what the model already considers plausible at that step.

This article builds a precise mental model for temperature in LLM sampling, works through a small numerical example, and explains how temperature interacts with deterministic decoding, truncation methods, and application requirements.

Start with logits, not words

Before selecting the next token, a language model produces a score for every token in its vocabulary. These raw scores are called logits. A softmax function converts them into probabilities.

For logits (z_1, z_2, \ldots, z_n), ordinary softmax assigns token (i) the probability

[ p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}. ]

Sampling then draws one token according to those probabilities.

Suppose a simplified model has only three possible next tokens:

Token       Logit
"red"       2.0
"blue"      1.0
"green"     0.0

Softmax turns these scores into probabilities of approximately:

Token       Probability
"red"       0.665
"blue"      0.245
"green"     0.090

"red" is the most likely token, but sampling can still choose either of the others. Temperature changes how strongly those logit differences influence the draw.

Temperature rescales the logits

With temperature (T), sampling commonly uses

[ p_i(T) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}, \qquad T > 0. ]

The division happens before softmax. That detail explains the behavior.

If T = 0.5, the example logits become:

[2.0, 1.0, 0.0] / 0.5 = [4.0, 2.0, 0.0]

Their differences get larger, so the distribution becomes sharper:

Token       Probability at T = 0.5
"red"       0.867
"blue"      0.117
"green"     0.016

If T = 2.0, the logits become:

[2.0, 1.0, 0.0] / 2.0 = [1.0, 0.5, 0.0]

The differences shrink, producing a flatter distribution:

Token       Probability at T = 2.0
"red"       0.506
"blue"      0.307
"green"     0.186

The ranking hasn’t changed: "red" remains above "blue", which remains above "green". What changes is the gap between their probabilities.

This gives a useful mental model:

  • temperatures below 1 sharpen the distribution;
  • a temperature of 1 leaves the logits at their original scale;
  • temperatures above 1 flatten the distribution.

These statements describe the standard positive-temperature softmax transformation. An API may expose a parameter called temperature with additional constraints or combine it with other decoding operations, so its documentation still defines the exact behavior you receive.

Temperature changes uncertainty nonlinearly

It’s tempting to assume that halving temperature simply doubles the probability of the most likely token. It doesn’t. Temperature rescales logits, and softmax then exponentiates and normalizes them.

The result depends on the differences between logits, not their absolute values. Adding the same constant to every logit leaves softmax probabilities unchanged. For example:

[2, 1, 0]
[12, 11, 10]

produce the same softmax distribution at the same temperature because their relative gaps are identical.

The existing shape of the distribution matters too. If two leading tokens have almost equal logits, lowering temperature may still leave both with meaningful probability. If one token is far ahead of every alternative, even a moderately high temperature may leave it dominant.

So a setting such as temperature = 0.7 doesn’t correspond to a fixed amount of randomness across prompts, models, or generation steps. It applies the same mathematical transformation to potentially very different logit distributions.

What happens near zero

The formula requires T > 0; division by zero is undefined. Mathematically, as positive temperature approaches zero, probability concentrates on tokens with the maximum logit.

If there is one unique maximum, the limiting behavior approaches greedy selection of that token. With exact ties for the maximum, the limit doesn’t identify one unique winner by itself.

This is why treating a literal temperature of zero as part of the formula is incorrect. Some inference APIs accept temperature = 0 as a special instruction for deterministic or greedy decoding, while others require a positive value or define the option differently. That is an API convention, not the softmax equation evaluated at zero.

Greedy decoding also differs conceptually from sampling at a small positive temperature. With any positive temperature and finite logits, lower-ranked tokens can retain nonzero probability. A random sampler can therefore choose them, even if that outcome is rare.

Higher temperature doesn’t create better ideas

Consider a coding assistant choosing a continuation after:

if user is None:

Suppose the model already assigns most probability to continuations that return an error, raise an exception, or handle an anonymous user. Increasing temperature can make lower-probability alternatives more likely to appear. That may increase variation among repeated generations.

But if the model assigns a bad continuation some probability, higher temperature can raise its chance too. Temperature doesn’t know which alternatives are clever, correct, secure, or relevant. It only changes their probabilities according to their logits.

The same reasoning applies to factual answers. Lower temperature can make repeated outputs more stable when one continuation dominates, but it doesn’t verify facts. A confidently wrong continuation may become even more dominant as the distribution is sharpened.

For applications where correctness matters, temperature belongs alongside evaluation, retrieval or tools when appropriate, validation, and explicit handling of model failures. It isn’t a substitute for those mechanisms.

A common way to describe the spread of a probability distribution is entropy:

[ H(p) = -\sum_i p_i \log p_i. ]

For a fixed finite set of logits under ordinary temperature-scaled softmax, increasing positive temperature makes the distribution no sharper and generally increases entropy until the distribution approaches uniform as temperature grows. Lowering temperature concentrates probability on the largest logits and reduces entropy, apart from cases such as tied or already-equal logits where the change can be flat.

That mathematical relationship is useful, but generation isn’t a single softmax draw. After a token is sampled, it becomes part of the context and changes the logits for the next step. One unusual early token can move the sequence into a very different region of possible continuations.

As a result, sequence-level diversity is an accumulated effect of many conditional decisions. You shouldn’t infer a precise change in whole-response diversity from temperature alone.

Temperature interacts with top-k and top-p sampling

Production decoders often combine temperature with a truncation method that removes some candidate tokens before sampling.

With top-k sampling, only the k highest-scoring candidates are retained. With top-p, or nucleus sampling, the decoder retains a smallest high-probability set whose cumulative probability reaches a configured threshold, subject to the implementation’s exact rules.

Temperature and truncation solve different problems. Temperature reshapes relative probabilities. Truncation changes which candidates are eligible for the final draw.

Imagine the model has a long tail of extremely unlikely tokens. Raising temperature can give more probability mass to that tail. A truncation rule can prevent much of the tail from entering the sample set at all.

There is an implementation detail worth checking before you reason numerically about a particular API: the order of temperature scaling, probability calculation, and truncation. Common generation stacks apply temperature before top-k or top-p filtering, but an API contract rather than a generic description should be your source of truth. The ordering matters especially for top-p because changing temperature can change how many tokens are needed to reach the cumulative threshold.

Don’t tune several decoding controls at once if you want to understand cause and effect. Changing temperature and top-p together may improve an evaluation score, but you won’t know which change produced the result or whether the combination is robust.

Choose temperature from the task, then measure it

There is no universal temperature that fits every application. The useful range depends on the model, prompt, other decoding settings, and what failure costs the application can tolerate.

For a structured extraction task, variation may have little value. If the system must return a schema-conforming answer from supplied text, deterministic decoding or a low-variance setup can simplify testing. Constrained decoding or schema validation may matter more than fine temperature adjustments.

For brainstorming, repeated samples are part of the product: users want meaningfully different candidates. A higher temperature may help expose alternatives that would rarely be selected from a sharper distribution, although excessive flattening can also surface weak continuations.

For an assistant that mixes factual questions and open-ended writing, one global setting may be a poor compromise. If the serving system permits it, decoding policy can be chosen by task class and evaluated separately.

A practical tuning loop is:

  1. Define the property you care about: correctness, schema validity, diversity, pass rate, human preference, or another measurable outcome.
  2. Hold the prompt, model version, and other decoding controls fixed.
  3. Evaluate several temperature settings on representative inputs.
  4. For stochastic settings, run enough repeated samples to observe variation rather than judging one output per prompt.
  5. Compare quality together with operational effects such as retries, validation failures, and output length.

Temperature itself doesn’t inherently make one forward pass more computationally expensive in a meaningful way; rescaling logits is small compared with model inference. But the chosen decoding behavior can change generated lengths, retry rates, and downstream validation, which can change end-to-end latency and cost.

Common temperature mistakes

Expecting a fixed seed to solve every reproducibility problem

A pseudorandom seed can make sampling reproducible only within the guarantees of the inference stack. Model changes, numerical kernels, batching, hardware, sampler implementations, or service-side behavior can affect reproducibility. If an API documents deterministic guarantees, rely on those documented conditions rather than assuming a seed alone is sufficient.

Treating low temperature as a factuality control

Sharpening a distribution favors tokens the model already scores highly. If those scores reflect a misconception or unsupported claim, lower temperature can produce a stable wrong answer. Measure factual behavior directly on the task you care about.

Copying a temperature across models

The same numeric temperature can produce different behavior because models can have different logit distributions and may be served with different decoding stacks. When changing models, retest the setting instead of treating it as a portable quality parameter.

Ignoring downstream constraints

If an output must parse as JSON, satisfy a grammar, select from a fixed label set, or call a tool with valid arguments, sampling diversity may be secondary to enforcing the allowed output space. Use structural constraints when the platform supports them and validate the result. Temperature alone cannot guarantee valid structure.

When temperature is the wrong lever

Temperature is useful when the problem is genuinely about how probability mass is distributed among candidate tokens. Many generation problems aren’t.

If answers omit required context, improve the context or prompt. If the model lacks current information, provide an appropriate retrieval or tool path. If output must obey a grammar, use constrained generation or validation where available. If a classifier needs trustworthy probability estimates, calibration is a different problem from decoding temperature.

Likewise, if latency is the bottleneck, lowering temperature doesn’t remove autoregressive decoding steps. Techniques such as caching, batching, quantization, or speculative decoding address different parts of the inference system.

A good diagnosis asks what needs to change before reaching for a sampling parameter.

Use temperature as a distribution control

The most useful way to reason about LLM sampling temperature is mechanical: divide logits by a positive temperature, apply softmax, then sample according to the resulting distribution. Lower values sharpen existing preferences; higher values flatten them. Neither operation decides whether a token is true or useful.

When tuning a real application, inspect temperature together with the rest of the decoding pipeline and evaluate it against the behavior users actually need. Once the metric is clear, temperature becomes a small, understandable control rather than a vague creativity setting.

Artificial Intelligence LLM Inference Sampling