https://nalar.dev/tags/distributed-training/ Recent content in Distributed Training on Nalar Hugo en-us Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/reduce-transformer-activation-memory-with-sequence-parallelism/ Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/reduce-transformer-activation-memory-with-sequence-parallelism/ <p>Large transformer training can run out of accelerator memory even after the model&rsquo;s weights are split across several devices. The reason is easy to miss: tensor parallelism can shard expensive matrix multiplications while some intermediate activations remain replicated on every worker in the tensor-parallel group.</p> <p><strong>Sequence parallelism</strong> removes part of that replication. For operations that work independently on each token, it partitions activations along the sequence dimension so each tensor-parallel worker keeps only a slice of the tokens. The workers temporarily reconstruct or reduce data where the tensor-parallel computation requires communication, then return to sequence-sharded activations.</p>