Tanh Logit Soft Capping Bounds Extreme Scores Before Softmax

A softmax can accept logits of any finite magnitude, but large score gaps make its output increasingly concentrated. Tanh logit soft capping inserts a bounded nonlinear transform before softmax so that no transformed logit exceeds a configured magnitude.

For a positive cap c, a common form is:

softcap(z; c) = c * tanh(z / c)

The operation does not clip at a hard threshold. It behaves almost linearly near zero and gradually compresses larger magnitudes as they approach -c or c.

The cap changes score gaps, not only absolute magnitude

Softmax is invariant to adding the same constant to every logit, so absolute logit magnitude alone does not determine the output distribution. Pairwise differences matter:

softmax(z)_i / softmax(z)_j = exp(z_i - z_j)

Applying a soft cap changes those differences whenever logits enter the nonlinear region. If s_i = softcap(z_i; c), then:

softmax(s)_i / softmax(s)_j = exp(s_i - s_j)

Because each transformed value lies strictly between -c and c for finite input, any transformed pair has a gap smaller than 2c. The resulting probability ratio is therefore bounded by the corresponding exponential gap for finite logits.

This is different from subtracting the maximum logit for numerical stability. Max subtraction preserves every pairwise difference and therefore preserves the mathematical softmax distribution. Soft capping intentionally changes the distribution when the transform is nonlinear.

Near zero, the transform is close to identity

The local behavior follows from the derivative:

d/dz [c * tanh(z / c)] = 1 - tanh(z / c)^2

At z = 0, the derivative is 1. For logits small relative to c, the transformed value stays close to the original value. As |z| / c grows, the derivative decreases toward zero and additional increases in magnitude produce progressively smaller changes after the cap.

This smooth transition distinguishes soft capping from hard clipping:

hardclip(z; c) = min(max(z, -c), c)

Hard clipping has an exact identity region and then a flat region beyond the threshold. Tanh soft capping starts compressing before reaching the asymptote and remains differentiable for finite inputs.

The distinction matters during training because the gradient through the transform is attenuated rather than abruptly set to zero at a finite boundary. It also matters at inference because a checkpoint trained with one transform represents a different function from the same parameters evaluated without it.

Ordering is preserved for finite logits

tanh is strictly increasing, and multiplication by a positive c preserves that ordering. For finite a and b:

a < b  =>  softcap(a; c) < softcap(b; c)

Soft capping therefore does not swap the rank of two finite logits. The token with the largest raw logit remains the token with the largest transformed logit when the same positive cap is applied elementwise.

Rank preservation does not mean probability preservation. Compression reduces large pairwise gaps, so the normalized probability mass can become less concentrated even though the ordering is unchanged.

That separation is useful when reasoning about greedy decoding. If greedy selection is based only on the argmax of a uniformly soft-capped logit vector, the selected index is unchanged in exact arithmetic. Sampling behavior can still change because sampling depends on the full normalized distribution rather than only its maximum index.

Temperature and soft capping do not generally commute

Temperature scaling divides logits by a positive temperature T before softmax:

softmax(z / T)

With soft capping present, operation order becomes part of the model definition. In general:

softcap(z; c) / T

is not equal to:

softcap(z / T; c)

The first expression compresses according to the original ratio z / c and then rescales the bounded result. The second changes the input to tanh, which changes the amount of compression itself.

A serving stack must therefore preserve the checkpoint’s specified order rather than treating temperature, soft capping, and softmax as interchangeable post-processing controls. Algebraic rearrangements that are valid for linear scaling can fail once a nonlinear transform is inserted.

The cap value sets the nonlinear scale

The parameter c controls both the output bound and the input scale at which compression becomes material. A larger c keeps a wider range of logits near the identity regime. A smaller c moves the same raw values farther into the saturating part of tanh.

This coupling means c is not merely a numeric safety threshold. Changing it modifies relative score gaps and therefore changes the probability distribution.

The limiting behavior makes the boundary visible. As c grows while a finite z stays fixed:

c * tanh(z / c) -> z

so the transform approaches the identity. At finite c, however, it remains part of the represented function.

Attention logits and vocabulary logits are different surfaces

A model can apply soft capping to attention scores, final vocabulary logits, both, or neither. These placements are not equivalent.

For attention, the transform acts before attention softmax and changes relative weighting among visible key positions. Masking semantics still need separate treatment; a masked position must remain excluded according to the architecture rather than being turned into an ordinary finite score by an inappropriate operation order.

For vocabulary logits, the transform acts on token scores before the output softmax or decoding rule. It changes the distribution over candidate tokens but does not retroactively alter hidden states already computed by the network.

A fused kernel may combine several operations, yet compatibility depends on preserving the same mathematical placement. Applying the right formula at the wrong surface produces a different model.

Numeric stability still requires a stable softmax

Bounding logits can limit the magnitude presented to a later softmax, but it does not replace a numerically stable softmax implementation. Standard stable evaluation commonly subtracts the row maximum before exponentiation:

s = softcap(z; c)
p_i = exp(s_i - max(s)) / sum_j exp(s_j - max(s))

The subtraction leaves the exact softmax result unchanged while reducing exponent range. Soft capping and max subtraction therefore serve different purposes: one intentionally reshapes score gaps, while the other is an algebraically equivalent way to evaluate softmax.

Precision, fused-kernel arithmetic, and special values such as infinities or NaNs also need explicit implementation rules. The finite-input bounds of tanh do not by themselves define behavior for every non-finite value encountered in a faulty or overflowing computation.

The implementation boundary is consequently narrow but strict. A soft cap is a model operation with a specified function, cap value, tensor surface, and order relative to masking, temperature, and softmax. Preserving those details keeps the deployed computation aligned with the function encoded by the checkpoint.