Fine-tuning a neural network on a new task can move parameters away from values that supported an earlier task. The new objective has no inherent reason to preserve those earlier behaviors when the earlier data is absent from the update. Elastic weight consolidation, commonly abbreviated EWC, adds a parameter-space constraint intended to reduce that drift.
The constraint is selective rather than uniform. Parameters estimated to be more consequential for the earlier task receive a larger penalty for moving, while parameters assigned lower importance can change more freely. That distinction is the central mechanism; EWC is not simply weight decay around zero.
The penalty is anchored to an earlier solution
Assume a model has parameters θ* after fitting task A. When adapting the same model to task B, EWC augments the task-B objective with a quadratic penalty:
L(θ) = L_B(θ) + (λ / 2) Σ_i F_i (θ_i - θ_i*)²Here, L_B is the objective for task B, θ_i* is the stored value of parameter i after task A, F_i is an importance estimate for that parameter, and λ controls the overall strength of the constraint.
The gradient contributed by the penalty for one parameter is
λ F_i (θ_i - θ_i*)so the restoring force grows with both displacement and estimated importance. A parameter with a near-zero F_i receives little resistance from this term. A parameter with a large F_i is pulled more strongly toward its earlier value as it moves away.
This differs from ordinary L2 regularization. Weight decay or an L2 penalty commonly favors smaller parameter magnitudes relative to zero. EWC favors proximity to a stored parameter vector, with a separate coefficient for each parameter.
Fisher information supplies the parameter weights
The original EWC formulation connects F_i to the diagonal of a Fisher information matrix evaluated around the parameters obtained for the earlier task. In probabilistic models, Fisher information describes sensitivity of the model’s log likelihood to parameter changes.
For a model distribution p(y | x, θ), a diagonal Fisher term can be expressed as an expectation of squared score components:
F_i = E[(∂ log p(y | x, θ) / ∂θ_i)²]with the expectation defined under the distribution used by the chosen Fisher formulation. In practice, implementations estimate these terms from a finite sample and retain only the diagonal. This turns a full quadratic form into independent per-parameter penalties and avoids storing a dense matrix whose size grows quadratically with the number of parameters.
That diagonal approximation also discards interactions between parameters. If two parameters can compensate for each other, a full curvature matrix can represent their coupled direction; a diagonal penalty cannot. The resulting importance values should therefore be treated as an approximation to local sensitivity, not as a complete decomposition of model behavior into independent parameters.
Implementation details around the Fisher estimate matter. Some code uses observed labels in squared log-likelihood gradients, while other formulations sample labels from the model distribution. These estimators need not produce identical values. Recording the estimator, data subset, reduction rule, and normalization is part of making an EWC configuration reproducible.
The regularization strength sets a stability boundary
The coefficient λ does not have a task-independent interpretation. Its effect depends on the scale of L_B, the scale of the Fisher estimate, optimizer settings, and any normalization applied to the penalty.
A very small coefficient makes adaptation resemble ordinary fine-tuning because the new-task gradient dominates. Increasing the coefficient restricts movement along coordinates with large F_i. If the constraint becomes too strong, the model may retain the earlier parameter configuration at the cost of fitting task B poorly.
This creates a measurable stability-plasticity trade. Evaluation should therefore retain separate metrics for the earlier and new tasks. A single aggregate score can conceal a configuration that protects one task while substantially degrading the other.
Loss reduction is another source of scale changes. If L_B changes from a sum to a mean over a batch while the EWC penalty is unchanged, the effective balance between the terms changes. The same numerical λ can then represent a different optimization problem.
Parameter importance depends on the reference task and point
EWC constrains movement relative to a particular parameter vector and a sensitivity estimate taken around that vector. Both are local artifacts of the earlier model state.
A parameter assigned low importance after task A is not globally unimportant. It may become central after later adaptation, or it may matter for inputs not represented in the sample used to estimate the Fisher diagonal. Likewise, a large Fisher entry describes local sensitivity under the selected distribution; it does not establish that the corresponding parameter has a unique semantic function.
This matters when tasks differ substantially. If task B requires movement in directions that task A marks as highly constrained, the objectives are directly in tension. EWC can alter the compromise but cannot create a parameter configuration that satisfies incompatible requirements.
The method also does not preserve earlier examples explicitly. Its retained state is a parameter reference plus importance values. Replay-based methods impose earlier behavior through stored or generated data instead, which constrains function outputs through additional examples rather than only through a local parameter penalty. The two mechanisms can therefore behave differently even when they consume similar extra memory.
Multiple earlier tasks require an explicit accumulation rule
After more than one prior task, a system must decide what the penalty represents. One option stores a separate reference and importance estimate for each earlier task, producing multiple quadratic terms. Storage then grows with the number of tasks because each term needs parameter-shaped state.
Another option consolidates importance information into a running representation and anchors the penalty to a more recent parameter vector. This bounds state growth but changes the approximation. The exact accumulation rule becomes part of the algorithm and should not be inferred from the label EWC alone.
Repeated Fisher estimates can also differ in magnitude across tasks because datasets, likelihoods, sample counts, or normalization differ. Adding them without a defined scaling policy can make later optimization dominated by whichever estimate happens to have the larger numerical scale.
A low penalty does not prove retained behavior
Parameter distance is only a proxy for functional preservation. Neural networks can be sensitive to small movement in some directions and insensitive to larger movement in others. EWC attempts to reflect that asymmetry through the Fisher weights, but the diagonal and finite-sample approximations leave gaps.
The direct check is behavioral: evaluate the earlier task after adaptation using the same decision rules and data protocol used for its baseline. The EWC penalty can be logged as an optimization diagnostic, but a small penalty value is not evidence by itself that earlier predictions remain acceptable.
The same boundary applies when comparing EWC with unconstrained fine-tuning. A smaller Euclidean parameter displacement does not establish stronger retention, and a larger displacement does not establish failure. The relevant comparison combines task metrics with the exact regularization setup that produced the checkpoint.
EWC is most interpretable as a local, weighted constraint on parameter movement. Its value comes from making the retention preference explicit in the objective without requiring the full earlier dataset. Its main limitation follows from the same design: preserving a quadratic neighborhood around one parameter state is only an approximation to preserving the model’s earlier function.