Neural network training does not usually move parameters smoothly toward one final point. Mini-batch gradients are noisy, learning-rate schedules change step sizes, and later updates can move a model between nearby parameter settings with noticeably different validation results.

That creates a practical question: should deployment use the parameters from one particular training step, or a smoothed version of several recent parameter states?

An exponential moving average, or EMA, provides the second option. During training, it maintains a separate copy of the model parameters that changes more slowly than the actively optimized parameters. The optimizer still trains the ordinary model. The EMA copy is typically used for evaluation or inference.

This article explains the mental model behind EMA weights, how the decay value controls their memory, what EMA does and does not guarantee, and how to evaluate it without accidentally comparing different training procedures.

Think of EMA as a second, slower model state

Suppose one scalar parameter takes these values after successive optimizer steps:

step 1: 1.00
step 2: 1.40
step 3: 1.10
step 4: 1.30

Using only the latest value means evaluation at step 4 sees 1.30. EMA instead combines the latest parameter with a running summary of earlier values.

For parameter vector theta_t at training step t, a common update is:

ema_t = beta * ema_(t-1) + (1 - beta) * theta_t

Here beta is the decay, with a value between 0 and 1. A larger decay gives more weight to history. A smaller decay makes the EMA follow the current training weights more closely.

For a deliberately small teaching example, let beta = 0.9, the previous EMA value be 1.20, and the new trained parameter be 1.40:

ema_new = 0.9 * 1.20 + 0.1 * 1.40
        = 1.22

The training parameter moved to 1.40, while the EMA parameter moved only from 1.20 to 1.22. That slower movement is the core idea.

Real models apply the same operation element by element across many parameter tensors. Production implementations should use framework-supported tensor operations rather than scalar loops.

EMA does not change the optimizer’s update

It is useful to separate two states:

training parameters -> receive gradients -> optimizer updates them
EMA parameters      -> receive no optimizer step -> track training parameters

In the usual setup, loss and gradients are computed from the training parameters. After the optimizer changes those parameters, the EMA state is updated from them. The EMA copy is not fed back into the optimizer on the next step.

This distinction prevents a common misconception. EMA is not itself an optimizer and does not make the gradients less noisy. It smooths the parameter state used for evaluation or inference.

A simplified training loop looks like this:

initialize model parameters
initialize EMA parameters from model parameters

for each training step:
    compute loss with model parameters
    compute gradients
    optimizer updates model parameters
    update EMA parameters from model parameters

The order matters. If the intended definition tracks parameters after each optimizer step, updating EMA before the optimizer silently tracks a different sequence of states.

Decay controls how quickly old updates disappear

Expanding the recurrence shows why EMA behaves like a weighted history. Ignoring initialization for a moment, recent parameter states receive weights proportional to:

current step:      (1 - beta)
1 step old:        (1 - beta) * beta
2 steps old:       (1 - beta) * beta^2
3 steps old:       (1 - beta) * beta^3
...

The weights shrink geometrically. EMA therefore does not keep a fixed window and then abruptly discard older states. Their influence fades continuously.

A useful rough scale for the averaging horizon is:

1 / (1 - beta)

For example:

beta = 0.9    -> scale of about 10 updates
beta = 0.99   -> scale of about 100 updates
beta = 0.999  -> scale of about 1000 updates

This is a rule of thumb, not an exact cutoff. An older state still has some influence beyond that many steps.

Another precise way to reason about memory is the half-life: the number of updates required for a contribution’s relative weight to fall by half.

half_life = log(0.5) / log(beta)

For beta = 0.999, the half-life is about 693 EMA updates. This makes one important implementation detail visible: decay is defined per EMA update, not inherently per training example, token, or epoch.

If one experiment updates EMA every optimizer step and another updates it every ten optimizer steps, using the same beta does not give the same time horizon.

Why averaging weights can help evaluation

Near a useful solution, stochastic training can move the model among nearby parameter states. A checkpoint taken at one step captures exactly one of those states. EMA reduces sensitivity to short-lived parameter movements by combining information from a sequence of states.

That can make validation behavior more stable, and in some training setups the EMA model achieves better evaluation quality than the final raw checkpoint. Neither outcome is guaranteed. Parameter averaging is most plausible when the states being averaged belong to a compatible region of parameter space.

EMA should therefore be treated as a candidate model state to measure, not as a universal quality improvement.

The safest comparison is straightforward:

same training run
    |-- evaluate current training weights
    `-- evaluate EMA weights

Using the same validation data and evaluation procedure isolates the effect of which parameter state is evaluated.

Choose the decay in training-step units

A decay value only becomes meaningful when connected to the update cadence and training length.

Suppose training has 50,000 optimizer steps. An EMA with an effective scale of roughly 1,000 updates can incorporate substantial recent history while still adapting during training. But if the entire run lasts only 200 steps, a very slow EMA may spend much of training dominated by its initialization.

Instead of copying a decay value from another project, ask:

  1. How many optimizer steps does this run contain?
  2. How often is EMA updated?
  3. How quickly should the averaged model respond when training enters a better region?
  4. Does validation improve across a reasonable range of decay values?

A high decay is not automatically better. If the training distribution, objective, or learning dynamics change during the run, a very slow EMA can retain stale parameter states longer than desired.

Handle initialization deliberately

The recurrence needs an initial EMA state. A simple approach is to copy the model parameters when EMA tracking begins:

ema_0 = theta_0

Early EMA values then contain substantial influence from that initial model. The effect fades as more updates arrive, but it can matter when decay is high or training is short.

Some implementations instead delay EMA tracking, vary the decay during early training, or use a bias-correction scheme. Those are implementation choices rather than requirements of EMA itself. If you compare experiments, record the initialization and update policy along with the decay value.

This is especially important when reproducing results. Two systems that both report beta = 0.999 can still produce different EMA states if one starts averaging immediately and the other starts thousands of steps later.

Keep non-parameter model state in mind

Not every value that affects inference is necessarily an optimized parameter. Some architectures or layers maintain buffers or other state, such as running statistics.

A generic statement like “use EMA weights” does not specify what should happen to those values. A framework utility may copy them, average them, or leave them to another update mechanism depending on its API and configuration.

For that reason, verify what your implementation tracks. The mathematical EMA rule for trainable parameters does not by itself define the correct handling of every model buffer.

This matters when switching between the training model and EMA model for validation: the evaluated state must be internally consistent, not merely a collection of averaged parameter tensors attached to unrelated buffers.

Account for memory and checkpointing costs

EMA needs another model-sized set of tracked values. That extra state can be significant for large neural networks.

If a model has N tracked parameters and the EMA copy stores them at the same precision, the EMA parameter storage is roughly another N values. Exact memory usage depends on data types, device placement, sharding, framework bookkeeping, and which tensors are tracked.

The extra arithmetic per update is simple, but moving or synchronizing a full parameter copy can affect training throughput in large distributed systems. Implementations may therefore update EMA less frequently, keep it on a different device, or integrate it with sharded training. Each choice changes either the effective averaging cadence or the systems cost, so benchmark the actual setup.

Checkpointing also needs an explicit policy. If deployment uses EMA weights, saving only the raw training parameters can discard the state you intended to serve. If training must resume with the same EMA trajectory, save both the training state and the EMA state together with the optimizer and scheduler state required by the training procedure.

Avoid common evaluation mistakes

The most damaging EMA mistakes are usually experimental rather than mathematical.

Comparing checkpoints from different data exposure

Do not compare an EMA model from a later training step with a raw model from an earlier step and attribute the difference to averaging. Evaluate both states from the same training point when testing the effect of EMA.

Selecting decay on the test set

Decay is a model-selection choice. If you try several values and choose the one with the best test result, the test set has influenced the model-selection process. Tune on validation data and reserve the test set for the final evaluation when that separation matters to your workflow.

Assuming smoother means calibrated

EMA may make model outputs or validation curves less variable, but that does not imply predicted probabilities are calibrated. Calibration is a separate property that must be measured with suitable data and metrics.

Forgetting the raw model

During development, retaining both raw and EMA evaluation results is useful. If EMA stops helping after a training change, the comparison can reveal it. Replacing the raw checkpoint silently removes that diagnostic signal.

When EMA is useful and when it is unnecessary

EMA is worth testing when neural network training produces noisy checkpoint-to-checkpoint validation behavior, when a training recipe is known to benefit from parameter averaging, or when deployment can afford the additional tracked state during training.

It is less compelling when the model is already stable and EMA shows no validation benefit, when training is so short that a slow average mostly reflects initialization, or when the extra model-sized state creates an unacceptable memory or systems cost.

EMA also does not replace ordinary checkpoint selection. If validation clearly identifies a strong checkpoint and averaging provides no measured advantage, the simpler checkpoint may be preferable.

Conclusion

An exponential moving average maintains a slower copy of neural network parameters while ordinary training continues unchanged. Its decay determines how quickly historical states lose influence, so the value must be interpreted together with update frequency and training length.

The practical workflow is simple: define when EMA starts, define when it updates, save the state you intend to deploy, and evaluate raw and EMA parameters at the same training point. If EMA improves the metrics that matter for your application at an acceptable systems cost, use it. If it does not, a normal checkpoint remains the simpler choice.

Artificial Intelligence Machine Learning Neural Networks Model Training