https://nalar.dev/tags/memory-optimization/ Recent content in Memory Optimization on Nalar Hugo en-us Wed, 16 Sep 2026 00:00:00 +0700 https://nalar.dev/trade-activation-memory-for-recomputation/ Wed, 16 Sep 2026 00:00:00 +0700 https://nalar.dev/trade-activation-memory-for-recomputation/ <p>Backpropagation needs intermediate values from the forward computation to form gradients. Retaining every required activation can consume substantial accelerator memory, especially as sequence length, batch size, hidden width, or network depth grows.</p> <p>Activation checkpointing changes that storage decision. Selected forward regions retain only chosen boundary tensors, then reproduce omitted intermediates when the backward pass reaches those regions. Peak activation memory can fall, but some forward computation is executed again.</p> <p>The useful engineering question is not simply whether checkpointing saves memory. The placement of recomputation boundaries determines which tensors disappear, how much extra compute appears, and whether replayed operations reproduce a valid backward computation.</p>