Logits on Nalar

Tanh Logit Soft Capping Bounds Extreme Scores Before Softmax

Thu, 24 Sep 2026 00:00:00 +0000

A softmax can accept logits of any finite magnitude, but large score gaps make its output increasingly concentrated. Tanh logit soft capping inserts a bounded nonlinear transform before softmax so that no transformed logit exceeds a configured magnitude.

For a positive cap c, a common form is:

copied = false, 1600)" class="absolute right-3 top-3 z-10 rounded-lg border border-line-strong bg-surface px-2 py-1 font-mono text-[0.8rem] text-muted opacity-0 transition group-hover:opacity-100 hover:bg-ink hover:text-white dark:border-night-line dark:bg-night dark:text-night-muted dark:hover:bg-white dark:hover:text-ink">

softcap(z; c) = c * tanh(z / c)

The operation does not clip at a hard threshold. It behaves almost linearly near zero and gradually compresses larger magnitudes as they approach -c or c.

Logit Bias Alters Token Odds Before Sampling

Wed, 23 Sep 2026 00:00:00 +0000

A token-level bias is usually applied to model scores before probabilities are normalized. That placement matters. Adding a constant to one token’s logit changes its odds relative to every token that does not receive the same constant, even though the model parameters and hidden state remain unchanged.

The mechanism is simple, but its operational effect depends on the rest of the decoding pipeline.

Additive bias acts on score differences

For a vocabulary with logits (z_1, \ldots, z_V), softmax assigns token (i) the probability

Repetition Penalty Rewrites Logits for Seen Token IDs

Wed, 23 Sep 2026 00:00:00 +0000

A repetition penalty can act before sampling by changing the logits of token IDs that already occur in a selected token history. The operation does not need to compare words, phrases, or rendered strings. Its unit can be the tokenizer’s integer ID, which gives the mechanism a narrower meaning than its name may suggest.

That distinction matters when a decoder emits subword tokens. Two strings that appear similar to a person can map to different token sequences, while a token reused inside unrelated words can still be marked as previously seen.