Optimization

Understand Gradient Noise in Mini-Batch Training

Neural network training usually updates model parameters from a small batch of examples rather than computing a gradient over the entire training set. That makes each update cheaper, but it also means the update direction depends on which examples happened to enter the batch. This variation is often called gradient noise. It is not necessarily a bug. It is a consequence of estimating a dataset-wide gradient from a sample, and it creates an important trade-off between computation per update, update variability, and training throughput.