Averaging two neural-network checkpoints can produce a useful parameter vector, yet leave BatchNorm running statistics tied to a different network. The weights define one set of activations; the stored running means and variances may describe activations produced by earlier weights. Inference then combines state from two different points in parameter space.

This mismatch is easy to miss because BatchNorm running statistics are buffers rather than trainable parameters in common implementations. A parameter-averaging routine can handle every weight correctly and still produce an internally inconsistent inference state.

BatchNorm state depends on the weights that generate activations

For one channel, BatchNorm at inference can be represented as:

y = gamma * (x - running_mean) / sqrt(running_var + eps) + beta

gamma and beta are trainable parameters. running_mean and running_var are state accumulated from training batches. The distinction matters after parameter averaging.

Suppose checkpoints A and B contain convolution weights W_A and W_B. A simple average creates:

W_avg = 0.5 * W_A + 0.5 * W_B

The activations produced by W_avg are not, in general, described by either checkpoint’s BatchNorm buffers. Averaging the two running means and variances also does not provide a general fix. Variance is nonlinear, and changing upstream weights can change both activation means and activation dispersion.

The problem is therefore not that the buffers were omitted from an arithmetic average. The deeper issue is that BatchNorm statistics characterize a distribution of activations generated by a particular network state.

Parameter averaging and state averaging are different operations

Weight averaging methods operate in parameter space. Stochastic weight averaging, checkpoint averaging, and related schemes can combine trainable tensors according to a chosen rule. BatchNorm running statistics have a different role: they estimate moments observed during forward execution.

Treating those buffers as ordinary parameters confuses two kinds of state. A weighted mean of parameters has a clear definition. A weighted mean of stored variances does not necessarily equal the variance of activations under the weighted model.

A simple identity exposes the issue. For random variables drawn from two distributions, total variance contains both within-distribution variance and a term from the difference between their means. Merely averaging two variance estimates drops that second contribution. With a newly averaged network, the activation distribution can differ further because the mapping from input to activation has changed.

This is also distinct from optimizer state. Momentum or adaptive optimizer accumulators affect future updates, but they are not normally consulted during inference. BatchNorm buffers directly affect inference outputs.

A statistics pass aligns buffers with the averaged network

A practical repair is to hold the averaged trainable parameters fixed and run representative inputs through the network so BatchNorm can rebuild its running statistics. No gradient update is required for this purpose. The objective is to observe activations from the model that will actually be evaluated.

The exact procedure depends on the framework’s BatchNorm semantics. The relevant conditions are consistent across implementations:

  • trainable parameters must remain unchanged during the statistics pass;
  • BatchNorm must update its running state from forward activations;
  • inputs should represent the distribution used for subsequent evaluation;
  • stochastic layers need deliberate handling so the collected activation distribution matches the intended model behavior.

The final point prevents a common ambiguity. Switching an entire model into a generic training mode may activate dropout as well as BatchNorm updates. That can make the collected statistics reflect randomly masked activations rather than the deterministic inference path. Framework-specific controls determine whether BatchNorm state can be refreshed without enabling unrelated stochastic behavior.

Momentum semantics affect the refresh

BatchNorm implementations commonly update a running statistic with a recurrence similar to:

running = (1 - m) * running + m * batch_stat

The symbol m here denotes the update weight for the current batch; API naming and exact conventions differ. Reusing old buffers with a small update weight can leave substantial influence from the pre-averaging model.

A full refresh should therefore be designed around the implementation’s documented accumulation rule. Some utilities reset the buffers and compute fresh estimates across a data pass. Other implementations require explicit control of momentum or counters. The desired invariant is more useful than a framework-specific recipe: the final running statistics should describe activations produced by the averaged parameters, not remain a mixture dominated by stale state.

Batch size also matters because each forward batch supplies a finite-sample estimate. Very small batches can make the refreshed moments noisy. A statistics pass does not remove BatchNorm’s dependence on representative sampling; it merely moves that estimation to the network state that will be used at inference.

The refresh dataset defines the operating distribution

Recomputing statistics on arbitrary data can replace one mismatch with another. If deployment inputs differ materially from the refresh data, the buffers describe the refresh distribution rather than the deployed one.

That makes the data choice part of the model artifact. A checkpoint containing averaged parameters plus refreshed BatchNorm state is tied to the input distribution used to estimate those buffers. If preprocessing changes, the activation moments can change even when the raw examples appear similar.

The refresh pass should also avoid target-dependent transformations or evaluation leakage. BatchNorm needs model inputs, not privileged labels. Using a held-out evaluation set to tune or select the resulting model would contaminate evaluation even if the forward pass itself does not use labels.

Validation should compare state, not only parameters

A parameter diff cannot confirm that an averaged model is ready for inference. Two checkpoints may have identical trainable tensors and different BatchNorm buffers, producing different outputs for the same input.

Useful validation therefore includes the non-parameter state that participates in inference. For BatchNorm networks, that means checking that running statistics are finite, were refreshed under the intended data path, and remain attached to the final averaged parameters. Output evaluation should happen after that state is finalized.

This boundary becomes especially relevant in pipelines that serialize only parameter tensors, swap averaged weights into a live module, or restore buffers from one constituent checkpoint. Those operations can all create a model that is numerically valid yet semantically mixed.

Weight averaging defines a new point in parameter space. For networks with BatchNorm, that point does not automatically come with matching activation statistics. Treating the statistics pass as part of constructing the averaged model keeps inference state aligned with the parameters that generate it.

Artificial Intelligence Neural Networks Batch Normalization Optimization