https://nalar.dev/tags/gradient-clipping/ Recent content in Gradient Clipping on Nalar Hugo en-us Thu, 24 Sep 2026 00:00:00 +0000 https://nalar.dev/global-gradient-norm-clipping-rescales-parameter-gradients-with-one-shared-factor/ Thu, 24 Sep 2026 00:00:00 +0000 https://nalar.dev/global-gradient-norm-clipping-rescales-parameter-gradients-with-one-shared-factor/ <p>Global gradient norm clipping treats the current parameter gradients as one aggregate vector. If its norm exceeds a threshold, every participating gradient is multiplied by the same coefficient. The operation changes update magnitude while preserving the direction of the aggregate gradient vector, apart from finite-precision effects.</p> <h2 id="one-norm-controls-the-whole-gradient-set">One norm controls the whole gradient set</h2> <p>Let the parameter gradients be <code>g_1, g_2, ... g_m</code>. For a p-norm, global clipping first forms the equivalent norm of their concatenation:</p>