https://nalar.dev/tags/optimization/ Recent content in Optimization on Nalar Hugo en-us Fri, 04 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-gradient-noise-in-mini-batch-training/ Fri, 04 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-gradient-noise-in-mini-batch-training/ <p>Neural network training usually updates model parameters from a small batch of examples rather than computing a gradient over the entire training set. That makes each update cheaper, but it also means the update direction depends on which examples happened to enter the batch.</p> <p>This variation is often called <strong>gradient noise</strong>. It is not necessarily a bug. It is a consequence of estimating a dataset-wide gradient from a sample, and it creates an important trade-off between computation per update, update variability, and training throughput.</p>