https://nalar.dev/tags/numerical-stability/ Recent content in Numerical Stability on Nalar Hugo en-us Mon, 14 Sep 2026 00:00:00 +0700 https://nalar.dev/bound-extreme-logits-with-soft-capping/ Mon, 14 Sep 2026 00:00:00 +0700 https://nalar.dev/bound-extreme-logits-with-soft-capping/ <p>A transformer can produce logits whose magnitudes grow far beyond the range needed to express a strong preference. Large attention scores can make a softmax distribution extremely concentrated, while large output logits can make token probabilities nearly one-hot. A hard clamp can bound those values, but it introduces a flat region with an abrupt derivative change at the threshold.</p> <p>Logit soft capping uses a smooth saturating function instead. One common form is:</p>