Optimizer updates can move model parameters back and forth even when the broader trajectory changes more gradually. An exponential moving average, or EMA, keeps a second parameter state that follows those updates with smoothing. The trainable model still receives ordinary optimizer updates; the EMA state is a derived copy used separately, often for evaluation or export.
The mechanism is compact, but its behavior depends on decay, update frequency, initialization, and which state is actually saved or evaluated.
EMA is a recursive parameter filter
Let theta_t denote the trainable parameters after optimizer update t, and let m_t denote the corresponding EMA parameters. With decay beta in the interval from zero up to, but not including, one:
m_t = beta * m_(t-1) + (1 - beta) * theta_tA larger beta gives more weight to prior EMA state and makes the shadow parameters respond more slowly. A smaller value tracks recent parameter updates more closely.
The recursion expands into a weighted history of parameter states. Ignoring initialization for a moment, contributions from older states decrease geometrically. A parameter state from k updates earlier receives a factor proportional to:
(1 - beta) * beta^kEMA therefore does not average a fixed-size window. Its history has no hard cutoff; influence decays with age.
At beta = 0, the shadow state simply copies the current parameters after every EMA update. Values close to one produce stronger smoothing but also a slower response to recent parameter movement.
Update order defines which state enters the average
An EMA update is usually associated with an optimizer update rather than with forward or backward computation. The order can be expressed as:
compute gradients
optimizer updates theta
EMA blends updated theta into mIf the EMA is updated before the optimizer instead, it incorporates the previous trainable state at that point. This creates an offset in the sequence of parameter states entering the recursion.
Gradient accumulation makes the distinction more visible. Suppose several microbatches contribute gradients to one optimizer update. Updating EMA after every microbatch would increase its update count without a corresponding new optimizer state. Updating it once after the accumulated optimizer update instead associates one EMA transition with one new trainable parameter state.
The decay coefficient only has meaning together with this cadence. The same numeric beta applied once per optimizer update and once per microbatch represents different smoothing over training progress.
Initialization leaves a transient
A direct implementation can initialize the shadow parameters from the initial trainable parameters:
m_0 = theta_0Subsequent EMA states then contain a geometrically decaying contribution from that initial copy. Another design can initialize an accumulator at zero and compensate for the resulting startup bias. These are different state definitions and should not be mixed silently.
With copied initialization, the recursive form after t updates includes a term proportional to beta^t * theta_0. For a large decay, that term can persist across many early updates. The effect follows directly from the recurrence rather than from optimizer momentum or gradient statistics.
Some systems vary the decay during an initial period so that the shadow state responds more quickly near the start. Such a schedule changes the weighting kernel over history; it is not equivalent to using one constant decay from the first update.
EMA parameters are not optimizer momentum
Both mechanisms contain exponentially weighted state, but they operate on different quantities.
Momentum-based optimizers maintain state derived from gradients or parameter updates and use that state to compute future trainable parameters. EMA model weights instead combine successive parameter values into a shadow copy. The shadow copy need not participate in gradient computation or determine the next optimizer update.
This separation matters when inspecting checkpoints. A checkpoint can contain trainable parameters, optimizer state, EMA parameters, or all three. Restoring only one component changes what can be resumed or evaluated.
Treating EMA weights as if they were optimizer state also obscures their deployment role. If inference is intended to use the shadow parameters, exporting only the current trainable parameters produces a different model state even though architecture and tensor shapes match.
Parameter selection must be explicit
A model can contain more state than trainable floating-point parameters. Examples include fixed parameters, integer counters, and framework-managed buffers. The EMA definition should specify which tensors participate.
For a simple parameter-only EMA, each included shadow tensor has the same shape as its source parameter and receives the same scalar decay. Non-floating state generally requires separate semantics because interpolation is not meaningful for values such as integer counters.
Shared parameters also need care. If two module paths refer to the same underlying parameter, an implementation should preserve the intended sharing rather than accidentally creating independent shadow values with conflicting update paths.
Device placement and numeric precision are implementation choices as well. Keeping shadow weights in reduced precision lowers storage cost but also changes rounding behavior in the recursive blend. Keeping them in a wider format consumes more memory. Neither choice changes the mathematical recurrence, but they can produce different stored values after many updates.
Evaluation must swap model state coherently
Using EMA weights for evaluation means the forward pass must see the complete intended shadow parameter set. Partially replacing parameters mixes states from different points in the smoothing process.
One approach keeps the trainable model untouched and loads EMA weights into a separate evaluation instance. Another temporarily swaps shadow values into the active model, runs evaluation, then restores the trainable values. The second approach requires careful restoration, especially if evaluation can fail midway or run concurrently with training.
Stateful buffers add another boundary. If the architecture has buffers updated during training, averaging only parameters does not automatically define corresponding averaged buffers. Evaluation code must follow the model’s actual state semantics rather than assuming that every tensor has an EMA counterpart.
Decay is tied to update count
A decay such as 0.999 is not a standalone description of smoothing. Its effective history is measured in EMA updates. If optimizer updates become less frequent because accumulation changes, the same decay spans a different amount of data or wall-clock time.
This is especially relevant when comparing runs with different batch construction or distributed update schedules. Matching the scalar decay does not necessarily match the weighting of parameter history relative to processed examples.
A useful specification records both the decay rule and the event that triggers an EMA update. That makes the parameter filter reproducible at the level that affects its behavior.
EMA model weights are best treated as a separate, versioned model state with explicit update semantics. The recurrence itself is simple; most integration errors come from attaching that recurrence to the wrong cadence, exporting the wrong state, or leaving the set of averaged tensors implicit.