Artificial Intelligence
24 Sep 2026
4 min read
Global Gradient Norm Clipping Rescales Parameter Gradients with One Shared Factor
Global gradient norm clipping treats the current parameter gradients as one aggregate vector. If its norm exceeds a threshold, every participating gradient is multiplied by the same coefficient. The operation changes update magnitude while preserving the direction of the aggregate gradient vector, apart from finite-precision effects. One norm controls the whole gradient set Let the parameter gradients be g_1, g_2, ... g_m. For a p-norm, global clipping first forms the equivalent norm of their concatenation: