A training configuration can contain a parameter named weight_decay without making it obvious what operation the optimizer actually performs. That ambiguity matters most with adaptive optimizers such as Adam: adding an L2 penalty to the loss and directly decaying parameters are not generally the same update.
The distinction is easy to miss because the two procedures are closely related under ordinary stochastic gradient descent (SGD). Once an optimizer rescales different coordinates using gradient history, however, the equivalence breaks.
This article develops the difference from a one-parameter example, shows why Adam changes the story, and explains the practical mental model behind AdamW. By the end, you should be able to read an optimizer update, identify where regularization enters it, and reason about weight_decay without relying on the option name alone.
Start with the purpose of shrinking weights
Neural networks can fit training data with many different parameter values. One common regularization idea is to discourage unnecessarily large weights. Two mechanisms are often discussed in this context:
- L2 regularization adds a penalty proportional to the squared parameter magnitude to the training objective.
- Weight decay directly shrinks parameters during an optimization step.
These descriptions sound almost interchangeable. For simple SGD, they can be made equivalent by choosing corresponding coefficients. That special case is the source of much of the terminology confusion.
To see the difference clearly, use one scalar parameter w and let L(w) be the original training loss.
L2 regularization changes the gradient
With L2 regularization, optimize a modified objective:
L_regularized(w) = L(w) + (lambda / 2) * w^2Its gradient is:
dL_regularized/dw = dL/dw + lambda * wIf plain SGD uses learning rate eta, the update becomes:
w_next = w - eta * (dL/dw + lambda * w)Rearrange it:
w_next = (1 - eta * lambda) * w - eta * dL/dwThe first term shrinks the existing parameter by a multiplicative factor. The second performs the ordinary gradient step.
Now compare that with an explicit weight-decay update:
w_next = (1 - eta * lambda) * w - eta * dL/dwFor this plain SGD update, they are the same equation. If you learned that L2 regularization and weight decay are equivalent, this is the setting in which that statement is justified.
The important condition is that SGD applies the gradient directly with one common learning-rate scaling. Adaptive optimizers do more to the gradient before updating the parameter.
Adam transforms the gradient before applying it
Adam maintains moving estimates derived from past gradients. In simplified notation, let g_t be the gradient at step t, m_t the first-moment estimate, and v_t the second-moment estimate. Ignoring details that are not needed for this comparison, an Adam-like update has the shape:
m_t = moving_average(g_t)
v_t = moving_average(g_t^2)
w_next = w - eta * m_hat_t / (sqrt(v_hat_t) + epsilon)The hats represent the bias-corrected estimates used by standard Adam. The key point is not the exact moving-average equations. It is that Adam transforms each gradient coordinate according to its accumulated history.
Now suppose an L2 penalty is added to the objective. The gradient given to Adam becomes:
g_t = dL/dw + lambda * wThat entire value enters Adam’s moment estimates. The regularization contribution lambda * w is therefore mixed into the same adaptive machinery as the training-loss gradient.
This is different from first computing an Adam update from dL/dw and then independently shrinking w.
A two-coordinate example exposes the difference
The distinction is easier to see with two parameters. Imagine that Adam’s accumulated history causes the current update to scale one gradient coordinate much more strongly than the other.
For a simplified teaching example, suppose the optimizer effectively applies these coordinate-wise multipliers before the common learning rate:
parameter A: 0.1
parameter B: 1.0Assume both parameters currently equal 10, and the L2 coefficient is 0.01. The L2 contribution to each raw gradient is therefore:
lambda * w = 0.01 * 10 = 0.1After the simplified adaptive scaling, those contributions become:
parameter A: 0.1 * 0.1 = 0.01
parameter B: 1.0 * 0.1 = 0.10Before the common learning rate is applied, the nominally identical L2 penalty has had different effective influence on the two coordinates because it passed through the adaptive transformation.
This example is deliberately simplified; real Adam scaling comes from its moment estimates rather than fixed multipliers. It demonstrates the important mechanism: when an L2 term is inserted into the gradient, an adaptive optimizer can transform that term differently across parameters.
Direct weight decay does not need to pass through that transformation.
AdamW decouples decay from the loss gradient
AdamW was introduced to separate weight decay from Adam’s gradient-based adaptive update. Conceptually, its parameter update can be read as two effects:
w_next = w
- adam_update_from_loss_gradient
- eta * lambda * wEquivalently, the decay portion alone multiplies the current parameter by approximately:
1 - eta * lambdafor a step using learning rate eta and decay coefficient lambda, under this common formulation.
The important word is decoupled. The loss gradient is used to update Adam’s moment estimates. The weight-decay term is applied separately rather than being added to that gradient before the adaptive transformation.
This preserves a clean mental model:
loss gradient -> adaptive Adam machinery -> optimization update
parameter value -------------------------> decay updateAdamW does not mean that weight decay becomes independent of every training choice. For example, the size of the decay applied per step in the formulation above still includes the current learning rate. Learning-rate schedules, number of optimization steps, parameter groups, and decay schedules can therefore affect the total shrinkage over training.
Do not infer semantics from the option name
A configuration field called weight_decay does not by itself tell you whether an implementation performs decoupled weight decay or adds an L2-like term to the gradient. Libraries and optimizer variants can use similar names for different update rules.
When correctness matters, inspect the optimizer documentation or update equation and answer three questions:
- Is the regularization term added to the loss or gradient?
- Does it enter momentum or adaptive moment estimates?
- Is parameter shrinkage applied separately from the gradient-based update?
If the decay is separate from Adam’s moment computation, the behavior follows the AdamW mental model. If lambda * w is added to the gradient before Adam processes it, the behavior is coupled L2 regularization instead.
This distinction is more useful than arguing about labels because the update equation determines the training behavior.
Decide which parameters should decay
Even after choosing decoupled weight decay, applying one coefficient indiscriminately to every trainable value may not match the intended experiment.
Large weight matrices are common decay targets. Some training recipes exclude other parameter types, such as bias vectors or parameters belonging to normalization layers. That is a modeling and optimization choice, not a universal law.
The practical requirement is to make parameter grouping explicit. A conceptual configuration might look like:
decayed group:
matrix weights -> weight_decay = 0.01
non-decayed group:
selected biases and normalization parameters -> weight_decay = 0The value 0.01 is only illustrative. A useful coefficient depends on the model, optimizer, learning-rate schedule, training duration, data, and evaluation target.
Also verify what your framework considers a parameter group. Accidentally placing a tensor in both groups, omitting a trainable tensor, or matching names too broadly can make the implementation disagree with the intended policy.
Tune decay together with the training schedule
Weight decay is not a knob that can be interpreted in isolation.
Consider the decoupled shrinkage factor for one step:
w_next = (1 - eta_t * lambda) * wIf the loss-gradient update were zero for several steps, repeated decay would multiply the parameter by a sequence of factors:
w_T = w_0 * product(1 - eta_t * lambda)This simplified expression makes an important dependency visible: changing the learning-rate schedule or the number of steps changes the accumulated shrinkage even when lambda stays fixed.
For small eta_t * lambda, each individual decay step may be tiny, but many steps can make the cumulative effect meaningful. Conversely, copying a decay coefficient from a training run with a very different schedule does not guarantee comparable regularization.
Treat learning rate, schedule, training duration, and weight decay as related hyperparameters. Compare configurations using held-out performance or another task-relevant evaluation rather than parameter norms alone.
Watch for common mistakes
Calling every L2 term weight decay
The names coincide in some SGD formulations, but the operations are not generally interchangeable with adaptive optimization. Describe the actual update when precision matters.
Adding L2 regularization on top of AdamW unintentionally
If an objective already contains an explicit L2 penalty and the optimizer also applies decoupled weight decay, both mechanisms act on the parameters. That may be intentional, but it is not the same experiment as using AdamW decay alone.
Comparing coefficients across different implementations
A numeric value has meaning only together with the update rule. Frameworks may also expose optimizer-specific conventions, parameter grouping, schedules, or defaults. Verify semantics before assuming that the same number produces the same regularization behavior.
Treating smaller weights as the final objective
Weight decay changes training dynamics in a way that can improve generalization for some tasks and settings, but a smaller parameter norm is not itself proof of a better model. Evaluate the behavior the model is supposed to provide.
Using decay to repair unstable optimization
Weight decay is a regularization mechanism, not a general cure for exploding gradients, unsuitable learning rates, numerical overflow, or poor data. If training is unstable, diagnose those causes directly instead of assuming stronger decay will fix them.
When a simpler approach is enough
If you use plain SGD without momentum or adaptive coordinate-wise scaling, the algebraic equivalence between an L2 penalty and multiplicative weight decay can make the distinction mostly a matter of coefficient convention.
Once momentum, adaptive preconditioning, schedules, or framework-specific optimizer behavior enters the picture, write down or inspect the actual update. That small step prevents a common category error: assuming a regularizer acts directly on parameters when it actually passes through optimizer state.
For Adam-family training, decoupled weight decay is useful when you specifically want the shrinkage operation separated from Adam’s gradient adaptation. That does not make AdamW universally preferable to every optimizer or regularization strategy. Optimizer choice still depends on the model, task, training budget, and empirical evaluation.
Keep the two paths separate in your mental model
The core distinction is where the regularization signal enters training.
With L2 regularization, the squared-weight penalty changes the objective, so its derivative becomes part of the gradient that the optimizer processes. With decoupled weight decay, parameter shrinkage is applied separately from the adaptive gradient transformation.
Plain SGD can hide this difference because the two updates can be algebraically equivalent after matching coefficients. Adam exposes it because gradient coordinates are transformed using optimizer state.
When you encounter weight_decay in a training configuration, do not stop at the name. Check the update rule, confirm which parameters receive decay, and evaluate the coefficient together with the learning-rate schedule and training duration. That is the practical path from a vague regularization setting to an optimizer behavior you can reason about.