Bound Extreme Logits with Soft Capping

A transformer can produce logits whose magnitudes grow far beyond the range needed to express a strong preference. Large attention scores can make a softmax distribution extremely concentrated, while large output logits can make token probabilities nearly one-hot. A hard clamp can bound those values, but it introduces a flat region with an abrupt derivative change at the threshold.

Logit soft capping uses a smooth saturating function instead. One common form is:

softcap(x, c) = c * tanh(x / c)

where c is a positive cap parameter. The result remains between -c and c, approaches the identity near zero, and gradually compresses larger magnitudes as they approach either bound.

The mechanism is simple, but its placement matters. Capping attention scores changes the distribution used to mix values inside the model. Capping final vocabulary logits changes the probability distribution used for token prediction. Those are separate interventions even when they use the same mathematical transform.

The transform is nearly linear near zero

For small x / c, the hyperbolic tangent is close to its input:

tanh(x / c) ≈ x / c

so:

c * tanh(x / c) ≈ x

This means logits that are small relative to the cap are changed only slightly. Compression becomes stronger as magnitude increases.

For example, with c = 10:

x = 1   -> softcap(x, 10) ≈ 1.00
x = 5   -> softcap(x, 10) ≈ 4.62
x = 10  -> softcap(x, 10) ≈ 7.62
x = 20  -> softcap(x, 10) ≈ 9.64

The transform does not preserve spacing among large logits. Values of 20 and 30 are both mapped close to the upper bound, so their difference becomes much smaller after capping.

That compression is the point. The cap limits how much extra separation an already-large raw logit can create.

Soft capping differs from hard clipping

A hard clamp can be written as:

hardcap(x, c) = min(c, max(-c, x))

Inside the interval, hard clipping leaves the value unchanged. Outside it, every larger magnitude maps exactly to a boundary.

Soft capping starts compressing before the nominal bound. Its derivative is:

d softcap(x, c) / dx = 1 - tanh(x / c)^2

At zero, the derivative is 1. As magnitude grows, the derivative moves toward 0 smoothly. Extreme raw logits therefore receive progressively smaller changes in the capped value.

Hard clipping has a different gradient profile. Its derivative is 1 inside the unclipped interval and 0 outside it, apart from the threshold points where the derivative is not defined in the ordinary sense. This creates a sharp transition between fully transmitted and fully suppressed changes.

Neither behavior is universally preferable. They encode different choices about how values near and beyond the threshold should behave.

Attention-score capping changes concentration

Scaled dot-product attention forms scores before softmax:

s_ij = q_i · k_j / sqrt(d)

If soft capping is applied, the softmax instead receives:

s'_ij = c * tanh(s_ij / c)

Softmax depends on score differences. Adding the same constant to every score leaves the distribution unchanged, but compressing large positive and negative scores changes those differences.

Consider two raw attention scores:

[20, 10]

Their difference is 10. With a cap of 10, the transformed values are approximately:

[9.64, 7.62]

and the difference is about 2.02. The resulting softmax is much less concentrated than the softmax over the original pair.

This effect depends on the entire score vector. If every score is small relative to the cap, attention remains close to the uncapped result. If only a few scores reach the saturating region, their advantage over competing positions is compressed.

The cap therefore acts on attention selectivity, not merely on numeric range. Changing it can alter which positions receive substantial weight.

Final-logit capping acts on token probabilities

A language model usually converts final vocabulary logits into probabilities with softmax. Applying soft capping immediately before that softmax limits the maximum separation that raw extreme logits can retain after the transform.

Suppose two token logits are:

[30, 10]

With a cap of 10, they become approximately:

[9.95, 7.62]

The original gap of 20 becomes about 2.33. Other vocabulary logits are transformed too, so the final probability shift depends on the complete vector.

This has a direct implication for decoding code. A runtime that omits a soft-cap operation specified by a model architecture is not computing the same probability distribution as the architecture definition. The practical effect may be small for logits that stay in the near-linear region, but equivalence cannot be assumed from the formula alone.

Temperature also interacts with placement. Applying temperature after soft capping computes:

softmax(softcap(logits, c) / T)

That is generally not equal to soft-capping already temperature-scaled logits:

softmax(softcap(logits / T, c))

because the nonlinear transform and division do not commute. An implementation should preserve the operation order defined by the model rather than treating the cap as an interchangeable sampling option.

The cap parameter sets a scale, not a hard activation point

The parameter c controls both the output bound and the scale at which compression becomes noticeable. A larger cap keeps a wider interval close to linear behavior. A smaller cap compresses moderate logits more strongly.

There is no single cap value implied by the transform itself. A value that is mild for one model can materially reshape scores in another model if their raw logit distributions differ.

This makes raw-logit inspection useful when reproducing an architecture. Useful diagnostics include the distribution of abs(x) / c, the fraction of values near the saturating region, and the difference between capped and uncapped softmax outputs. Those measurements connect the configured cap to the actual values produced by the model.

Attention and final output layers also need not share a cap. Their logits serve different roles and can occupy different numeric ranges. Treating one cap as a global model constant can silently change behavior when the architecture specifies separate values.

Kernel support can change architectural fidelity

Fused attention kernels often combine score computation, masking, softmax, and value aggregation into one optimized operation. An architecture-specific transform between score computation and softmax requires the kernel to expose or implement that extra operation.

If a selected kernel does not support soft capping, replacing it with an uncapped attention path is an architectural change, not just a performance setting. The same issue applies to export formats and inference engines that recognize standard attention patterns but omit model-specific score transforms.

A compatibility check should therefore compare operations, not only tensor shapes and parameter names. Loading every weight successfully does not prove that the runtime reproduces the model’s computation.

For final vocabulary logits, implementation is usually simpler because the logits are already materialized before sampling. Even there, the cap must occur before probability conversion and in the intended numeric precision.

Saturation can hide raw-logit growth

A bounded output does not imply bounded pre-cap values. Once raw logits enter the saturating region, substantially larger inputs can produce only small changes after the transform.

That matters for diagnostics. Monitoring only capped logits can make raw-score growth difficult to see. During model development, retaining statistics from both sides of the transform can distinguish a stable capped distribution from increasingly extreme pre-cap activations.

The distinction also matters when changing or removing a cap. A model whose raw logits regularly sit far into saturation can react much more strongly than one whose logits rarely approach the cap scale.

Soft capping is therefore best treated as part of the model’s computation graph rather than a cosmetic numeric guard. Its smooth bound changes score geometry in a predictable way, and the size of that change depends on where the model’s logits sit relative to the configured cap.