https://nalar.dev/tags/gradient-checkpointing/ Recent content in Gradient Checkpointing on Nalar Hugo en-us Fri, 04 Sep 2026 00:00:00 +0700 https://nalar.dev/reduce-training-memory-with-gradient-checkpointing/ Fri, 04 Sep 2026 00:00:00 +0700 https://nalar.dev/reduce-training-memory-with-gradient-checkpointing/ <p>Training a neural network can run out of accelerator memory even when the model parameters fit comfortably. The missing piece is often <strong>activations</strong>: intermediate values produced during the forward pass and retained because backpropagation needs them later.</p> <p>Gradient checkpointing, also called activation checkpointing, trades extra computation for lower activation memory. Instead of keeping every intermediate activation until the backward pass, training keeps selected checkpoints and recomputes missing forward values when their gradients are needed.</p>