https://nalar.dev/tags/batch-normalization/ Recent content in Batch Normalization on Nalar Hugo en-us Fri, 18 Sep 2026 00:00:00 +0000 https://nalar.dev/recompute-batchnorm-statistics-after-weight-averaging/ Fri, 18 Sep 2026 00:00:00 +0000 https://nalar.dev/recompute-batchnorm-statistics-after-weight-averaging/ <p>Averaging two neural-network checkpoints can produce a useful parameter vector, yet leave BatchNorm running statistics tied to a different network. The weights define one set of activations; the stored running means and variances may describe activations produced by earlier weights. Inference then combines state from two different points in parameter space.</p> <p>This mismatch is easy to miss because BatchNorm running statistics are buffers rather than trainable parameters in common implementations. A parameter-averaging routine can handle every weight correctly and still produce an internally inconsistent inference state.</p>