Temperature is often exposed as a single generation parameter, but its effect is narrower than a general control for output quality. For a fixed vector of finite logits, positive temperature rescales the gaps before softmax. It changes the resulting probabilities without changing which logit is larger than another.
That distinction matters when a serving layer combines temperature with greedy selection, top-k filtering, top-p filtering, penalties, or implementation-specific handling of zero temperature. The same numeric setting can participate in a different decoding pipeline even though the underlying scaling operation is simple.
Positive temperature preserves logit ordering
Given logits (z_i) and temperature (T > 0), the scaled distribution is
[ p_i(T) = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}. ]
For any two finite logits (z_a) and (z_b), division by the same positive value preserves their order:
[ z_a > z_b \quad \Longrightarrow \quad z_a/T > z_b/T. ]
Softmax is monotonic in each scaled logit when the others are fixed, so the token with the largest finite logit remains the token with the largest probability after positive temperature scaling. Ties also remain ties under this operation.
Probability ratios show the part that does change:
[ \frac{p_a(T)}{p_b(T)} = \exp\left(\frac{z_a-z_b}{T}\right). ]
A smaller positive (T) magnifies a fixed logit gap in this ratio. A larger (T) compresses it. Temperature therefore changes probability concentration, not the rank induced by the original logits.
The limiting cases have different semantics
As (T) approaches zero from the positive side, probability mass concentrates on the maximum-logit set. With one unique maximum, the limiting distribution places all mass on that token. With several equal maxima, the mathematical limit distributes mass among those tied maxima rather than selecting one through a tie-breaking rule.
This limit does not make literal division by zero valid. An inference API that accepts temperature = 0 must define its own behavior. Some systems use that value as a signal for deterministic decoding; others may reject it or apply a separate code path. That behavior belongs to the implementation contract, not to the softmax formula above.
At the other extreme, as (T) grows without bound, finite scaled logits approach zero. If softmax is applied over a fixed finite candidate set with no other masking or filtering changes, the distribution approaches uniform over that set. Real decoding pipelines can alter the candidate set before sampling, so this limiting statement applies only to the stated operation order.
Greedy decoding does not gain variability from temperature
If a decoder applies positive temperature scaling and then chooses argmax directly, the selected token is unchanged because positive scaling preserves logit order. Temperature only affects the probabilities that would be used by a sampling operation or by later processing that depends on their values.
This is a useful boundary for API design. A parameter can be accepted syntactically yet have no effect on token selection under a pure greedy path. Treating temperature as inherently stochastic confuses probability transformation with the decision rule that consumes the transformed values.
The distinction also applies to reproducibility. Sampling from a temperature-scaled distribution introduces a random draw unless the sampling implementation is otherwise deterministic under a fixed state. Greedy argmax has no such draw. Temperature alone does not specify which path is active.
Filtering order can change the resulting candidate set
Top-k filtering depends on rank. Since positive temperature scaling preserves rank, applying temperature before or after selecting the top (k) logits gives the same membership when ties are handled consistently. The probabilities assigned inside that retained set still depend on temperature.
Top-p filtering is different because its cutoff depends on cumulative probabilities. Temperature changes those probabilities, so applying temperature before top-p can change how many tokens fall inside the retained probability mass. Reversing the order is not generally equivalent.
Penalties and logit biases add another distinction. If an implementation modifies logits before temperature scaling, the modified gaps are divided by (T). If it applies a transformation after scaling, the effective magnitude relative to the original logits can differ. Parameter names alone are insufficient to establish equivalence between serving stacks; operation order is part of the decoding semantics.
Numerical implementation still needs stable softmax
Temperature does not remove the usual numerical concerns around exponentiation. A stable implementation commonly subtracts the maximum scaled logit before exponentiation:
[ p_i = \frac{\exp(z_i/T - m)}{\sum_j \exp(z_j/T - m)}, \qquad m = \max_j(z_j/T). ]
Subtracting the same (m) from every scaled logit leaves the softmax distribution unchanged while reducing overflow risk. Very small positive temperatures can create large scaled gaps, so finite precision and special values still require explicit handling in production code.
Masked logits, infinities, NaNs, quantized kernels, and fused decoding operators can introduce behavior outside the simple finite-logit derivation. Those cases should be treated according to the runtime’s documented numerical contract rather than inferred from the ideal formula.
The practical boundary is compact: positive temperature is a monotonic rescaling of finite logits before softmax. It can substantially change sampling probabilities while leaving logit rank intact. Any change beyond that—candidate filtering, deterministic zero-temperature behavior, penalties, or token selection—comes from the surrounding decoding pipeline.