https://nalar.dev/tags/training/ Recent content in Training on Nalar Hugo en-us Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/stabilize-neural-network-training-with-gradient-clipping/ Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/stabilize-neural-network-training-with-gradient-clipping/ <p>Neural network training can look healthy for many steps and then suddenly become unstable. The loss may jump, parameters may receive an unusually large update, or numerical values may become non-finite. One possible cause is an <strong>exploding gradient</strong>: the gradient becomes large enough that the resulting optimization step is destructive.</p> <p><strong>Gradient clipping</strong> puts a limit on gradients before the optimizer uses them. It is especially useful when occasional gradient spikes are expected, but it is not a general repair for a bad learning rate, broken data, or an incorrect training loop.</p>