Numerical Stability

Bound Extreme Logits with Soft Capping

A transformer can produce logits whose magnitudes grow far beyond the range needed to express a strong preference. Large attention scores can make a softmax distribution extremely concentrated, while large output logits can make token probabilities nearly one-hot. A hard clamp can bound those values, but it introduces a flat region with an abrupt derivative change at the threshold. Logit soft capping uses a smooth saturating function instead. One common form is: