Training a neural network often becomes memory-bound before it becomes compute-bound. You may want a batch of 64 examples for stable optimization, but the model, activations, optimizer state, and input tensors leave enough accelerator memory for only 8 examples at a time.
Reducing the batch size to 8 may work, but it also changes the optimization process. Gradient accumulation provides another option: process several smaller microbatches, add their gradients together, and update the model only after the desired effective batch has been processed.
The technique is simple, but a few details matter. Loss scaling, optimizer steps, learning-rate schedules, gradient clipping, and stateful layers can all change the result if they are handled at the wrong point.
The mental model: separate processing size from update size
A normal training step with a batch of 32 examples looks roughly like this:
32 examples -> forward pass -> backward pass -> optimizer stepIf only 8 examples fit in memory, gradient accumulation can divide that logical batch into four microbatches:
8 examples -> forward/backward -> keep gradients
8 examples -> forward/backward -> keep gradients
8 examples -> forward/backward -> keep gradients
8 examples -> forward/backward -> keep gradients
|
v
optimizer stepThe model processes only 8 examples at once, so activation memory is based on the microbatch rather than the full logical batch. The parameter gradients remain allocated and receive contributions from each backward pass.
For equal-sized microbatches, a useful relationship is:
effective batch size = microbatch size * accumulation steps * data-parallel workersFor example, a microbatch size of 8 accumulated for 4 steps on one worker gives an effective batch size of 32. With 2 data-parallel workers using the same settings, the global effective batch is 64, assuming each worker processes different examples.
This distinction is the key idea: microbatch size controls how much data is processed simultaneously, while effective batch size controls how much data contributes to an optimizer update.
Why gradients can be accumulated
Backpropagation computes a gradient for each model parameter. For a batch whose loss is the mean of per-example losses, the desired gradient is the mean of the per-example gradients.
Suppose a logical batch is divided into four equal microbatches. If each microbatch produces a mean loss L_i, then the logical batch mean is:
L = (L_1 + L_2 + L_3 + L_4) / 4Differentiation is linear, so its gradient is:
grad(L) = (grad(L_1) + grad(L_2) + grad(L_3) + grad(L_4)) / 4That is why the common implementation divides each microbatch loss by the number of accumulation steps before calling backward. Each backward pass adds its scaled gradient to the gradients already stored on the parameters.
A minimal training loop
The framework details vary, but the control flow can be expressed in Python-like pseudocode:
accumulation_steps = 4
optimizer.zero_grad()
for step, batch in enumerate(loader, start=1):
output = model(batch.inputs)
loss = loss_fn(output, batch.targets)
loss = loss / accumulation_steps
loss.backward()
if step % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()There are three important details here.
First, zero_grad() is not called after every microbatch. Doing that would erase the gradients you intended to accumulate.
Second, optimizer.step() runs once per logical batch, not once per microbatch.
Third, dividing the loss by accumulation_steps keeps the accumulated gradient on the scale of a mean loss over the full logical batch when the microbatches are equal in size.
Without that division, the accumulated gradient is approximately the sum of the microbatch mean gradients. That makes it larger by the accumulation factor compared with the corresponding full-batch mean gradient. You could compensate elsewhere, but scaling the loss makes the intended relationship explicit and is usually easier to reason about.
Handle a final partial accumulation window
The minimal loop has a bug if the number of loader batches is not divisible by accumulation_steps: the final gradients never reach an optimizer step.
A better control flow also updates on the final microbatch:
optimizer.zero_grad()
for step, batch in enumerate(loader, start=1):
output = model(batch.inputs)
loss = loss_fn(output, batch.targets)
(loss / accumulation_steps).backward()
update_due = step % accumulation_steps == 0
final_batch = step == len(loader)
if update_due or final_batch:
optimizer.step()
optimizer.zero_grad()However, there is another subtlety. If the final window contains fewer than accumulation_steps equally sized microbatches, dividing each loss by the original accumulation count makes that final update smaller than the mean gradient for the examples actually present.
You have several valid choices depending on the training design:
- drop the incomplete final window;
- construct batches so every accumulation window is complete;
- scale the final window by its actual number of microbatches;
- weight losses by example count when microbatch sizes vary.
The last option is the most general because the mathematically correct full-batch mean depends on the number of examples, not merely the number of backward calls.
Gradient accumulation does not reproduce every property of a large physical batch
It is tempting to say that accumulation is exactly the same as training with a larger batch. That is true only under appropriate conditions.
For operations where each example is independent during the forward and backward passes, accumulated gradients can match the gradient from the corresponding full batch apart from normal floating-point effects. But some model behavior depends on which examples are processed together.
Batch-dependent layers can behave differently
Batch normalization computes statistics from the current physical batch during training. Four forward passes of 8 examples therefore do not generally behave like one forward pass of 32 examples, even if their gradients are accumulated before the update.
Many modern transformer architectures use normalization methods that do not compute statistics across the batch dimension, so this particular issue may not apply. The broader lesson remains: check whether any operation couples examples within a physical batch.
Random operations need not match exactly
Dropout and other stochastic operations draw random values during each forward pass. Accumulation can still be a perfectly valid training method, but you should not expect bit-for-bit equivalence with a single large forward pass.
Losses can depend on other examples in the batch
Contrastive objectives and other batch-coupled losses may compare examples with one another. Splitting a batch into microbatches changes which examples can interact inside the loss, so ordinary gradient accumulation does not automatically reproduce the objective of the larger physical batch.
Put update-dependent operations at the optimizer boundary
Once you distinguish microbatches from optimizer updates, several training operations become easier to place correctly.
Learning-rate schedules
A scheduler defined in terms of optimizer steps should normally advance when the optimizer advances, not after every microbatch. Otherwise a four-step accumulation factor can make the schedule progress four times faster relative to parameter updates.
When configuring warmup or total training steps, count the units expected by the scheduler. Libraries differ, so verify whether their APIs define a step as a dataloader iteration or an optimizer update.
Gradient clipping
If you want to clip the gradient for the effective batch, clip after all gradients in the accumulation window have been collected and before the optimizer step:
accumulate -> accumulate -> accumulate -> accumulate
-> clip gradients
-> optimizer stepClipping every microbatch separately changes the resulting gradient because clipping is nonlinear. The clipped sum is not generally equal to the clipping of the final accumulated gradient.
Mixed-precision training
Mixed-precision systems may use loss scaling to protect small gradients from underflow. Gradient accumulation is compatible with this approach, but unscaling, overflow checks, clipping, and optimizer updates must follow the mixed-precision framework’s expected order.
Do not assume that manually dividing a loss for accumulation replaces the framework’s loss-scaling mechanism. They solve different problems: accumulation scaling controls the logical batch average, while mixed-precision loss scaling addresses numerical range.
What memory does gradient accumulation actually save?
The main saving comes from activations and other tensors whose size grows with the number of examples processed simultaneously. Reducing the physical microbatch can reduce their peak memory footprint.
Gradient accumulation does not make the model parameters disappear. Parameter storage, gradient buffers, and optimizer state still consume memory. For large models, optimizer states can be a major part of the memory budget.
That means accumulation helps most when batch-dependent activation memory is the limiting factor. If the model and optimizer state barely fit even with a microbatch of one example, accumulation alone cannot solve the problem. Techniques such as lower-precision training, activation checkpointing, parameter-efficient fine-tuning, or distributed sharding address different parts of the memory budget.
The throughput trade-off
Gradient accumulation reduces peak memory, but it does not magically make the work cheaper. The same examples still require forward and backward computation.
A smaller physical batch can sometimes use accelerator hardware less efficiently because matrix operations have less parallel work. On the other hand, accumulation may enable a model or sequence length that would otherwise run out of memory. The useful comparison is therefore not simply “accumulation versus no accumulation,” but the set of configurations that actually fit on the target hardware.
Measure both training throughput and memory usage. The largest microbatch that fits comfortably is often a reasonable starting point, then accumulation can be used to reach the desired effective batch size.
Do not copy a batch size without reconsidering the learning rate
Changing the effective batch size changes how frequently parameters are updated for a fixed number of training examples. It can also change gradient noise and optimization behavior.
Rules that scale the learning rate with batch size are useful in some training regimes, but they are not universal guarantees. The right learning rate depends on the optimizer, model, task, schedule, and training setup.
If accumulation is used only to reproduce an existing effective batch size, keeping the rest of a known-good optimization configuration is a sensible baseline. If you intentionally increase the effective batch size, treat the learning rate and schedule as hyperparameters that may need reevaluation.
Common mistakes
A few mistakes account for many confusing accumulation results:
- Clearing gradients every microbatch. This prevents accumulation entirely.
- Stepping the optimizer every microbatch. This changes the effective batch and optimization path.
- Forgetting loss normalization. This changes gradient scale when microbatch losses are means.
- Ignoring incomplete windows. The final examples may be skipped or incorrectly weighted.
- Advancing update-based schedules per microbatch. The learning-rate timeline no longer matches optimizer updates.
- Clipping each microbatch independently. The final direction and magnitude can differ from clipping the accumulated gradient.
- Assuming exact equivalence with a physical large batch. Batch-dependent or stochastic operations can behave differently.
When debugging, log the microbatch count, optimizer-step count, effective batch size, learning rate, and gradient norm. Those values quickly reveal many boundary errors.
When gradient accumulation is a good fit
Use gradient accumulation when the effective batch you want does not fit in memory as one physical batch and the model’s computation can be safely divided into microbatches.
It is especially useful when activation memory grows with batch size or sequence length and you can accept the throughput characteristics of smaller microbatches.
Do not reach for it automatically when a smaller effective batch already trains well. Extra accumulation adds state and boundaries to the training loop. It is also not a substitute for techniques that reduce parameter or optimizer-state memory when those are the actual bottleneck.
Conclusion
Gradient accumulation separates two quantities that are easy to confuse: how many examples fit through the model at once and how many examples contribute to one parameter update.
The reliable pattern is to process memory-safe microbatches, accumulate correctly scaled gradients, and perform optimizer-dependent operations only at the logical update boundary. Then account explicitly for incomplete windows and any computation that depends on the physical batch.
Used with those constraints in mind, gradient accumulation is a practical way to train with a larger effective batch without requiring that entire batch to fit in accelerator memory at once.