A decoder can assign the same ranking to every token before and after temperature scaling while producing substantially different probabilities. The mechanism is simple: for positive temperature T, logits are divided by T before softmax. Division by the same positive scalar preserves order, but softmax converts the changed gaps between logits into a different probability distribution.
That distinction matters in inference systems because temperature does not select tokens by itself. Its visible effect depends on what happens after the scaled softmax: direct sampling, top-k filtering, top-p filtering, greedy selection, or another decoding rule.
Positive temperature preserves logit ranking
Let the model produce logits z_i for vocabulary items i. Temperature scaling forms
p_i(T) = exp(z_i / T) / sum_j exp(z_j / T)for T > 0.
If z_a > z_b, then z_a / T > z_b / T. The exponential function is strictly increasing, and the softmax denominator is shared across all tokens. As a result, p_a(T) > p_b(T) for every finite positive T.
Temperature therefore cannot reorder unequal logits under this definition. Tied logits remain tied. This is a mathematical property of uniform positive scaling, not a claim about an entire decoding pipeline. A system that applies token-specific penalties, masks, biases, truncation, or other logit processors can change ordering before or after the temperature operation depending on its implementation.
Probability ratios expose the actual change
The ratio between two token probabilities removes the softmax denominator:
p_a(T) / p_b(T) = exp((z_a - z_b) / T)This expression isolates the effect of temperature. For a fixed positive logit gap, a smaller T increases the probability ratio, while a larger T moves the ratio toward 1.
Consider logits 3 and 1. Their gap is 2:
T = 0.5 -> ratio = exp(4)
T = 1.0 -> ratio = exp(2)
T = 2.0 -> ratio = exp(1)The higher-logit token remains higher in all three cases. What changes is the relative mass assigned to it compared with the lower-logit token.
This pairwise view also avoids a common ambiguity in phrases such as “more random.” Temperature changes the categorical distribution presented to a sampler. The realized token is still a random draw only when the subsequent decoding procedure actually samples from that distribution.
Greedy decoding can make temperature numerically irrelevant to selection
Pure greedy decoding chooses the token with the largest score. If temperature is the only transformation applied and T is finite and positive, the argmax is unchanged because the logit ranking is unchanged.
That means changing temperature alone does not change the selected token in an idealized greedy decoder. Implementations sometimes expose a temperature parameter even when a greedy mode bypasses sampling, so API configuration should not be treated as proof that temperature participates in token selection.
The statement also has boundaries. Numerical behavior at extreme values, special handling for T = 0, and the ordering of other logit processors are implementation details. Many libraries treat zero temperature as a request for deterministic decoding rather than literally evaluating division by zero. That convention is an API decision, not part of the softmax formula above.
Top-k and top-p react differently
Top-k keeps a fixed number of highest-ranked candidates. Since positive temperature scaling preserves rank, applying top-k after temperature leaves the identity of the top k tokens unchanged, assuming no other score transformation intervenes. Their normalized probabilities inside that set can still change.
Top-p behaves differently because its boundary depends on cumulative probability mass rather than a fixed rank count. A sharper distribution can reach the probability threshold with fewer tokens; a flatter distribution can require more tokens. The exact candidate count depends on the logits, the threshold, and implementation details such as whether filtering occurs before or after a particular normalization step.
Temperature can therefore alter a top-p candidate set without ever changing the ranking of the logits. This is a useful separation between rank effects and mass effects: top-k is primarily rank-bounded, while top-p is probability-mass-bounded.
The limit cases clarify the mechanism
As T approaches zero from the positive side, differences in scaled logits grow. Probability mass concentrates on the maximum-logit entries. With a unique maximum, its probability approaches 1. With an exact tie among maxima, the limiting mass is shared among those tied entries rather than selecting one through temperature scaling alone.
As T grows without bound, each finite logit divided by T approaches zero. Over a finite unmasked candidate set, the softmax distribution approaches uniform probability across those candidates.
These are mathematical limits. Production decoders operate at finite numeric precision, may mask parts of the vocabulary, and may clamp or reject parameter ranges. The limits describe the transformation, not an API guarantee for extreme parameter values.
Temperature does not repair the model’s score semantics
A softer or sharper softmax does not add information to the logits. It applies a one-parameter transformation to their relative gaps. If a model assigns a high score to an unsuitable token, temperature does not identify that token as unsuitable; it only changes how strongly score differences affect the resulting probabilities.
The same boundary matters when interpreting token probabilities as confidence. Generation logits are conditional scores over the next-token vocabulary under the model and current context. Adjusting temperature changes those probabilities mechanically. It does not, by itself, establish calibration against an external notion of correctness.
For serving systems, temperature is best treated as one stage in a decoding contract. Its mathematical effect is determined by positive scalar scaling before softmax, while observable generation behavior also depends on masks, penalties, truncation rules, sampling, random state, and the order in which the runtime applies them. Two endpoints with the same temperature value need not behave identically if the rest of that contract differs.