Softmax on Nalar

Additive Logit Bias Changes Token Odds Before Sampling

Wed, 23 Sep 2026 00:00:00 +0000

A decoder can favor or suppress a token without changing model weights. Add a constant to that token’s logit before softmax, and its probability changes relative to the rest of the vocabulary. The operation is simple, but its effect depends on where the bias enters the decoding pipeline and on every transformation that follows it.

This makes additive logit bias useful as an inference control, but not as a general semantic constraint. It changes a score used by the decoder. It does not rewrite the model’s internal representation or guarantee that a concept disappears from generated text.

Attention Logit Scaling Keeps Dot Products in a Stable Softmax Range

Wed, 23 Sep 2026 00:00:00 +0000

A dot product between a query and a key tends to grow in magnitude as their dimension grows. In scaled dot-product attention, the score is divided by the square root of the key dimension before softmax. That factor is not a cosmetic normalization. It controls the scale presented to softmax under a specific statistical assumption about the query and key components.

The familiar expression is

copied = false, 1600)" class="absolute right-3 top-3 z-10 rounded-lg border border-line-strong bg-surface px-2 py-1 font-mono text-[0.8rem] text-muted opacity-0 transition group-hover:opacity-100 hover:bg-ink hover:text-white dark:border-night-line dark:bg-night dark:text-night-muted dark:hover:bg-white dark:hover:text-ink">

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

where d_k is the query-key dimension for one attention head. The scaling term affects the distribution of attention probabilities even though it does not change the ordering of logits by itself.

Temperature Scaling Changes Softmax Sharpness Without Changing Logit Order

Wed, 23 Sep 2026 00:00:00 +0000

A decoder can assign the same ranking to every token before and after temperature scaling while producing substantially different probabilities. The mechanism is simple: for positive temperature T, logits are divided by T before softmax. Division by the same positive scalar preserves order, but softmax converts the changed gaps between logits into a different probability distribution.

That distinction matters in inference systems because temperature does not select tokens by itself. Its visible effect depends on what happens after the scaled softmax: direct sampling, top-k filtering, top-p filtering, greedy selection, or another decoding rule.