Neural network parameters do not move smoothly toward a final solution. Stochastic optimization updates them using noisy minibatch gradients, so the weights used after one training step can differ slightly from those used after the next. Saving only the final step therefore makes one particular point on that training path responsible for evaluation and deployment.
An exponential moving average, or EMA, keeps a second copy of the parameters that changes more gradually. Instead of replacing this copy with every new set of training weights, each update blends the previous average with the current parameters.
For some training setups, evaluating this smoothed copy can produce more stable or better validation results than evaluating the latest training weights. It is not guaranteed to help, and it does not replace good optimization or validation. This article explains what EMA weights represent, how to update them correctly, and the implementation details that most often cause misleading results.
Keep training weights and EMA weights separate
Imagine a model with one scalar parameter. The normal optimizer produces these values over several steps:
step 1: 1.00
step 2: 1.20
step 3: 0.90
step 4: 1.10The parameter moves because each minibatch produces a different gradient. An EMA maintains another value alongside it:
training parameter: updated by optimizer
EMA parameter: updated from training parameterThe optimizer continues to train the ordinary parameters. The EMA copy is usually not optimized by backpropagation. It follows the training parameters according to a smoothing rule and is used later for evaluation, checkpointing, or inference.
This separation is essential. EMA is not a different gradient optimizer; it is a way to average parameter states over time.
The EMA update
For a model parameter theta, let theta_ema be its moving-average copy. After an optimizer update, a common EMA rule is:
theta_ema = decay * theta_ema + (1 - decay) * thetaThe decay value lies between 0 and 1.
With decay = 0.9, the new EMA contains 90% of the previous EMA value and 10% of the current training parameter. With decay = 0.999, it changes much more slowly.
Consider a simplified example with an initial EMA value of 1.00 and decay = 0.9. If the optimizer changes the training parameter to 1.20, then:
theta_ema = 0.9 * 1.00 + 0.1 * 1.20
= 1.02If the next optimizer update changes the training parameter to 0.90:
theta_ema = 0.9 * 1.02 + 0.1 * 0.90
= 1.008The training parameter moved from 1.20 to 0.90, while the EMA moved only from 1.02 to 1.008. That slower movement is the point.
Why recent weights matter more
Expanding the recurrence shows why this is called an exponential moving average. Ignoring initialization for a moment, the contribution of older parameter values is repeatedly multiplied by decay.
Conceptually:
current weight -> contribution proportional to (1 - decay)
one update older -> contribution proportional to decay * (1 - decay)
two updates older -> contribution proportional to decay^2 * (1 - decay)
...Older states never receive a hard cutoff, but their influence shrinks geometrically. A larger decay gives the average a longer memory; a smaller decay makes it track recent training weights more closely.
This differs from a simple average over a fixed window. EMA needs only the current average and current model parameters, so its extra state is roughly one additional parameter copy rather than a history of checkpoints.
Update EMA after the optimizer step
Order matters. The EMA should represent the sequence of parameter states produced by optimizer updates.
A simplified training loop is:
for each batch:
loss = model(batch)
compute gradients
optimizer step
update EMA from current model parameters
clear gradients as required by the training loopIf you update EMA before the optimizer step, the average observes the previous parameter state instead of the newly produced one. That one-step shift may seem minor, but it makes the implementation differ from the intended update rule and can interact with initialization or schedules.
With gradient accumulation, update EMA when an actual optimizer step occurs, not after every microbatch that only contributes gradients. Otherwise the same unchanged parameter state may be averaged repeatedly between optimizer updates, changing the effective smoothing behavior.
The same principle applies in distributed training: define EMA updates in terms of the model state after the logical synchronized optimizer update rather than in terms of how many workers or input batches happened to participate.
Decay is measured in optimizer updates
A decay value has no useful interpretation without an update frequency.
Suppose one training setup performs 1,000 optimizer updates per epoch and another performs 10,000. Using the same decay in both does not give the EMA the same memory when measured in epochs or examples.
A useful way to reason about the timescale is the approximate effective window:
1 / (1 - decay)This is a rule of thumb, not a hard window. For example:
decay = 0.9 -> about 10 updates
decay = 0.99 -> about 100 updates
decay = 0.999 -> about 1,000 updatesThe exponential weighting still includes states older than that estimate. The approximation simply helps compare decay choices with the number of optimizer updates in a training run.
If training lasts only a few hundred updates, an extremely large decay can cause the EMA to retain too much influence from initialization. If training lasts millions of updates, a small decay may track the ordinary weights so closely that little smoothing occurs.
Initialization affects early EMA values
The recurrence needs an initial theta_ema. A straightforward choice is to copy the model’s parameters when EMA tracking begins:
EMA parameters = current model parametersAfter that, apply the normal update after each optimizer step.
Another implementation might initialize the accumulator differently and use a bias correction or a changing early decay. Those are legitimate variants, but they are not interchangeable with simple copy initialization. When reproducing a training recipe, check how the EMA state starts rather than copying only the headline decay value.
This matters most early in training. With a high decay, the initial EMA state can influence the average for many updates.
Evaluate the EMA copy without corrupting training
During validation, the model must use one coherent parameter set. A common pattern is:
save current training parameters
load or swap in EMA parameters
evaluate
restore training parametersAnother design keeps a separate model object containing EMA parameters. Either can work if state is managed correctly.
The important requirement is that training resumes with the optimizer’s current parameters, not with EMA parameters accidentally substituted into the optimizer state. Optimizers such as momentum-based methods also maintain internal state associated with the training trajectory. Swapping averaged parameters into training without a deliberate algorithmic reason can break the relationship between those parameters and optimizer state.
Treat EMA as a parallel evaluation state unless the training algorithm explicitly specifies otherwise.
Parameters are not the whole model state
Some neural network layers contain state that is not updated by gradients. Batch normalization, for example, can maintain running statistics used during inference.
An EMA implementation that averages trainable parameters but ignores other inference-relevant state needs a clear policy for that state. Depending on the framework and training recipe, you may keep the current running statistics, maintain corresponding averaged state where appropriate, or recompute statistics before final evaluation.
There is no universal rule that every buffer should be exponentially averaged. Some buffers are counters or discrete values for which arithmetic averaging is meaningless. Others have update semantics different from learned parameters.
Before implementing a generic average every tensor in state_dict approach, classify the state you are averaging. EMA should operate on quantities for which the averaging operation makes sense.
EMA can help without changing training loss
The ordinary model parameters still determine forward passes used to compute the training loss, unless the algorithm explicitly says otherwise. The EMA copy follows those parameters but does not normally feed back into gradient computation.
As a result, training loss can look identical whether EMA tracking is enabled or disabled, while validation results from EMA weights differ from validation results from current weights.
That is expected. EMA changes which parameter state you evaluate; it does not necessarily change the optimization path.
This also means you should log the two evaluation modes distinctly:
validation/current_weights
validation/ema_weightsIf metrics are recorded under one name while silently alternating parameter sets, comparisons become difficult to interpret.
Do not assume smoothing guarantees improvement
EMA can reduce sensitivity to short-term parameter fluctuations, but several conditions can make it neutral or harmful.
If the model is still moving rapidly toward a better region, a long-memory EMA can lag behind the current parameters. Near the beginning of training, this lag can be substantial.
If a learning-rate schedule deliberately moves the model through qualitatively different regions, averaging too much history may combine states that are not useful together. The EMA operates in parameter space; it does not know which earlier states should be forgotten for task-specific reasons.
If training is already very stable, EMA may provide little measurable benefit. And if poor validation quality comes from label errors, distribution mismatch, underfitting, or an unsuitable objective, smoothing parameters does not fix the root cause.
Use validation evidence to decide whether the averaged copy is useful.
EMA is different from checkpoint averaging
Checkpoint averaging and EMA share the idea of combining parameter states, but their weighting and storage differ.
A simple checkpoint average might take the arithmetic mean of several saved models:
average = (checkpoint_1 + checkpoint_2 + checkpoint_3) / 3Each selected checkpoint has equal weight. EMA updates continuously and gives more weight to recent states.
Checkpoint averaging can be applied after training if the required checkpoints were saved. EMA usually requires tracking the running state during training, unless the full sequence of parameter states is available later.
Neither method should be confused with an ensemble. An ensemble evaluates multiple models separately and combines their predictions. Parameter averaging first creates one parameter set and then performs one model evaluation. That distinction affects both behavior and inference cost.
Checkpoint both states when training must resume
A robust checkpoint for EMA training should preserve enough information to continue the same process. That commonly includes:
training model parameters
optimizer state
learning-rate scheduler state, if any
EMA parameters
EMA update count or schedule state, if usedSaving only EMA weights may be enough for inference, but it is not necessarily enough to resume training faithfully. Saving only ordinary model weights loses the moving average accumulated so far.
When loading an older checkpoint that has no EMA state, choose an explicit policy. For example, initialize EMA from the restored training parameters and document that the average restarts at that point. Do not silently invent a historical EMA that the checkpoint does not contain.
Measure the real trade-off
EMA requires an additional copy of the averaged model state. For a large model, that memory can be significant. The update also adds memory traffic and arithmetic after optimizer steps.
The inference cost is different. If you deploy only EMA weights, inference usually needs one model copy and one forward pass, just as it would with ordinary weights. You do not need to keep both parameter sets in the serving process unless the application uses both.
This makes EMA attractive when its validation benefit is real: it can change the deployed parameter choice without requiring ensemble-style multiple forward passes. But the training-time memory overhead can still matter, especially for models already close to accelerator memory limits.
Common implementation mistakes
Several mistakes can make EMA results confusing:
- Updating after every microbatch. If parameters change only on accumulated optimizer steps, this changes the intended averaging timescale.
- Updating before the optimizer. The EMA tracks stale parameter states relative to the stated recurrence.
- Forgetting initialization semantics. High decay can preserve a poor initial accumulator for many updates.
- Averaging every model buffer blindly. Not all non-parameter state has meaningful arithmetic averaging semantics.
- Failing to restore training weights after validation. Training may continue from EMA parameters with optimizer state built for a different trajectory.
- Saving only one state when resumability matters. The optimizer model and EMA model serve different purposes.
- Comparing decay values without update frequency.
0.999means something different across training loops with very different numbers of optimizer steps. - Assuming EMA must improve accuracy. Its value is empirical and depends on the optimization trajectory and task.
When to use EMA weights
EMA is worth testing when validation metrics fluctuate late in training, stochastic optimization produces noticeably noisy checkpoints, or a proven training recipe for the model family includes parameter averaging.
It is particularly convenient when you want a smoothed evaluation model but do not want the inference cost of running an ensemble.
Skip it when the extra training-state memory is unacceptable, when training is too short for the chosen averaging timescale, or when validation shows no meaningful benefit. If the real problem is unstable or divergent optimization, investigate learning rate, gradients, data, and numerical behavior rather than treating EMA as a substitute for fixing training.
Conclusion
Exponential moving average weights maintain a slowly changing copy of neural network parameters alongside the parameters updated by the optimizer. The method is simple: after each real optimizer step, blend the current weights into the EMA state using a decay factor.
Using EMA correctly requires more than applying the formula. The decay must be interpreted in optimizer-update units, initialization must be explicit, inference-relevant non-parameter state needs a deliberate policy, and checkpoints must preserve both training and EMA state when resumability matters.
Evaluate ordinary and EMA weights separately on the same validation data. If the averaged copy improves the decisions or metrics that matter enough to justify its training-time cost, it can provide a practical deployment model without adding multiple inference passes.