Batch Normalization

Recompute BatchNorm Statistics After Weight Averaging

Averaging two neural-network checkpoints can produce a useful parameter vector, yet leave BatchNorm running statistics tied to a different network. The weights define one set of activations; the stored running means and variances may describe activations produced by earlier weights. Inference then combines state from two different points in parameter space. This mismatch is easy to miss because BatchNorm running statistics are buffers rather than trainable parameters in common implementations. A parameter-averaging routine can handle every weight correctly and still produce an internally inconsistent inference state.