https://nalar.dev/tags/mixed-precision/ Recent content in Mixed Precision on Nalar Hugo en-us Thu, 10 Sep 2026 00:00:00 +0700 https://nalar.dev/prevent-fp16-gradient-underflow-with-dynamic-loss-scaling/ Thu, 10 Sep 2026 00:00:00 +0700 https://nalar.dev/prevent-fp16-gradient-underflow-with-dynamic-loss-scaling/ <h1 id="prevent-fp16-gradient-underflow-with-dynamic-loss-scaling">Prevent FP16 Gradient Underflow with Dynamic Loss Scaling</h1> <p>Mixed-precision training can reduce memory use and accelerate supported operations, but <code>float16</code> introduces a numerical problem that is easy to miss: some gradients are too small to survive in FP16. They can round to zero before the optimizer gets a chance to use them.</p> <p><strong>Loss scaling</strong> addresses that problem by multiplying the loss before backpropagation, which multiplies the resulting gradients by the same factor. The gradients are divided by that factor before the optimizer update, so the intended update is unchanged when the arithmetic remains finite. <strong>Dynamic loss scaling</strong> adjusts the factor during training so you don&rsquo;t have to guess one fixed value for the whole run.</p>