Training

Stabilize Neural Network Training with Gradient Clipping

Neural network training can look healthy for many steps and then suddenly become unstable. The loss may jump, parameters may receive an unusually large update, or numerical values may become non-finite. One possible cause is an exploding gradient: the gradient becomes large enough that the resulting optimization step is destructive. Gradient clipping puts a limit on gradients before the optimizer uses them. It is especially useful when occasional gradient spikes are expected, but it is not a general repair for a bad learning rate, broken data, or an incorrect training loop.