Large transformer training can run out of accelerator memory even after the model’s weights are split across several devices. The reason is easy to miss: tensor parallelism can shard expensive matrix multiplications while some intermediate activations remain replicated on every worker in the tensor-parallel group.
Sequence parallelism removes part of that replication. For operations that work independently on each token, it partitions activations along the sequence dimension so each tensor-parallel worker keeps only a slice of the tokens. The workers temporarily reconstruct or reduce data where the tensor-parallel computation requires communication, then return to sequence-sharded activations.
This article builds the idea from a small example, explains why it is usually paired with tensor parallelism, and shows how to reason about memory savings, communication, compatibility, and the cases where another technique is a better fit.
Start with the memory problem, not the name
Assume one training sample has a sequence length of 4096 tokens and hidden size 8192. Ignore the batch dimension for the moment.
A hidden-state tensor has shape:
[4096 tokens, 8192 hidden values]If those values use two bytes each, one such tensor contains:
4096 * 8192 * 2 bytes = 64 MiBTraining does not keep only one hidden-state tensor. Backpropagation may need inputs or intermediate values from many layers and operations. The exact saved tensors depend on the architecture, kernels, precision, and checkpointing policy, but the important scaling fact is simple: many activation costs grow with sequence length.
Now place the model on four tensor-parallel workers. Splitting weight matrices across four devices does not automatically imply that every activation is one quarter as large. Some activations can still have the full [4096, 8192] logical shape on every worker.
If an activation is replicated, four workers collectively store four copies. More importantly for avoiding an out-of-memory error, each worker still pays the full 64 MiB for that tensor.
Sequence parallelism asks a narrower question:
When an operation treats tokens independently, why should every tensor-parallel worker keep every token’s activation?
The core mental model: shard tokens between communication points
Suppose the four workers partition the 4096-token sequence evenly:
worker 0: tokens 0..1023
worker 1: tokens 1024..2047
worker 2: tokens 2048..3071
worker 3: tokens 3072..4095Each worker now holds a local activation with shape:
[1024 tokens, 8192 hidden values]At two bytes per value, that local tensor is 16 MiB rather than 64 MiB. For activations that remain sequence-sharded, the per-worker storage falls roughly with the sequence-parallel group size.
This does not mean the entire transformer can run on isolated sequence chunks. Self-attention is an obvious counterexample: a token may need information from tokens outside its local chunk. Tensor-parallel linear layers also have their own partitioning and reduction requirements.
The useful pattern is therefore not “split the sequence once and never communicate.” It is:
sequence-sharded activations
|
| gather when a tensor-parallel region needs the full representation
v
tensor-parallel computation
|
| reduce and scatter results back across the sequence dimension
v
sequence-sharded activationsThe sequence partition is a memory layout between communication boundaries. It does not change which tokens the mathematical model is allowed to use.
Why tensor parallelism creates the opportunity
Tensor parallelism splits work inside a layer. A common transformer implementation partitions large linear transformations across a group of workers. Depending on how a linear layer is partitioned, workers may need collective communication to combine partial results.
Without sequence parallelism, a collective reduction can leave the same complete result on every worker. That replication is useful for the next operation, but it can be wasteful when that next operation is token-wise.
Consider a simplified path:
parallel linear -> combine partial outputs -> dropout -> residual -> normalizationDropout, residual addition, and normalization such as LayerNorm are normally applied independently at each token position. They do not need token 20 on the same worker as token 3000 merely to perform those operations.
Instead of reducing partial outputs and replicating the complete result everywhere, a sequence-parallel implementation can use a reduce-scatter. Conceptually, reduce-scatter does two things:
- sums the partial results that must be combined;
- gives each worker only its assigned slice of the combined output.
The following token-wise operations then run on those local slices.
Before the next tensor-parallel region needs a different layout, an all-gather can reconstruct the required activation across the group.
A useful identity for the mental model is:
all-reduce = reduce-scatter + all-gatherAn ordinary all-reduce combines values and leaves the combined result on every participant. Sequence parallelism can separate those phases: reduce-scatter now, keep the result sharded while possible, and all-gather later when the replicated form is actually required.
This is why sequence parallelism is especially natural inside an existing tensor-parallel group. It reuses communication boundaries that already exist while avoiding unnecessary activation replication between them.
A four-worker example
Imagine that a tensor-parallel operation produces partial results for an activation Y. Each worker has contributed to every token, so those partial values must be summed.
With a conventional all-reduce:
worker 0 --\
worker 1 ---- sum partial Y -> full Y on workers 0, 1, 2, 3
worker 2 ----/
worker 3 --/All four workers can now run token-wise operations, but all four also store the complete Y.
With reduce-scatter:
worker 0 --\
worker 1 ---- sum partial Y -> tokens 0..1023 on worker 0
worker 2 ----/ tokens 1024..2047 on worker 1
worker 3 --/ tokens 2048..3071 on worker 2
tokens 3072..4095 on worker 3The numerical reduction is still performed. What changes is where the combined result lives afterward.
If the next tensor-parallel computation later requires the full sequence representation on each participant, an all-gather reverses the layout transition:
four token slices -> all-gather -> full sequence representationFor backpropagation, the corresponding collective operations must preserve the same mathematical gradients. Production frameworks implement custom distributed operators or autograd rules for these transitions; simply slicing a tensor in application code is not an equivalent implementation.
What sequence parallelism saves
The main benefit is lower activation memory per tensor-parallel worker for tensors that can stay partitioned along the sequence dimension.
Let an activation have logical shape:
[B, S, H]where:
Bis micro-batch size;Sis sequence length;His hidden size.
If it is replicated across a tensor-parallel group of size T, each worker stores roughly B * S * H elements for that activation. If the same activation is evenly sequence-sharded, each worker stores roughly:
B * (S / T) * HFor that tensor, the idealized per-worker reduction factor is therefore T.
That factor must not be applied to the entire training memory budget. Parameters, gradients, optimizer states, temporary workspaces, attention intermediates, communication buffers, and activations inside regions that cannot remain sequence-sharded have different layouts. Real peak-memory savings are consequently smaller than “all memory divided by T.”
This distinction matters when capacity planning. Measure peak allocated memory for the actual model and training configuration rather than extrapolating from one activation tensor.
Sequence parallelism is not context parallelism
The names are similar enough to cause configuration mistakes.
The sequence parallelism described here is tied to tensor parallelism and primarily avoids replicated activations around token-wise parts of transformer layers. A worker may temporarily gather the representation needed by tensor-parallel computation.
Context parallelism is a broader long-context technique. It partitions the sequence across workers through the network, including the attention workload. Because self-attention for one query can depend on keys and values from remote sequence chunks, context-parallel attention needs an explicit way to exchange the required key/value information or partial attention results.
A practical distinction is:
sequence parallelism:
shard selected activations between tensor-parallel communication points
context parallelism:
distribute the long sequence itself, including attention workFramework terminology is not perfectly universal, so confirm what a configuration flag means in the framework you use. Do not assume any option containing the word sequence implements the mechanism in this article.
Compare it with activation checkpointing
Sequence parallelism and activation checkpointing solve the same high-level problem—activation memory pressure—but use different resources.
Activation checkpointing saves memory by discarding selected forward activations and recomputing them during backward. Its central trade-off is:
less stored activation memory <-> more computationSequence parallelism saves memory by distributing selected activations across workers that already cooperate through tensor parallelism. Its central trade-off is closer to:
less replicated activation memory <-> distributed layout and collective communication constraintsThey can be combined. If sequence parallelism removes enough replication, a workload may need less aggressive checkpointing and therefore less recomputation. If activations are still too large, checkpointing can reduce them further.
Neither technique is automatically preferable. On a single accelerator, sequence parallelism across a tensor-parallel group is not available, while checkpointing can still help. On a multi-GPU job that already uses tensor parallelism, sequence parallelism may recover memory that would otherwise be duplicated across those workers.
Practical trade-offs to evaluate
Memory savings depend on what is actually sharded
Do not estimate the benefit from sequence length alone. Inspect the framework’s implementation and memory profile. If the dominant peak allocation comes from a tensor that sequence parallelism does not shard, enabling it may not solve the out-of-memory problem.
Likewise, fused kernels can change which intermediates are materialized and retained. Two implementations of the same mathematical transformer may have different activation-memory profiles.
Communication still has a cost
Replacing a replicated layout with reduce-scatter and later all-gather does not make communication disappear. In implementations that decompose communication already required by tensor parallelism, sequence parallelism can avoid adding the volume of a completely separate synchronization pattern, but latency, topology, collective efficiency, and overlap with computation still affect runtime.
A configuration that saves memory can therefore have neutral, positive, or negative throughput impact depending on the implementation and hardware. Benchmark end-to-end step time instead of assuming the memory optimization is free.
The group size constrains token partitioning
An even partition is simplest when the relevant sequence dimension is divisible by the sequence-parallel group size. Frameworks may pad, use uneven partitions, or impose divisibility requirements. Padding itself can waste computation, so verify the framework’s behavior rather than relying on the idealized arithmetic above.
Distributed correctness is more than tensor shape correctness
The forward values and backward gradients must cross layout boundaries consistently. A tensor with the expected local shape can still be mathematically wrong if a reduction, gather, or gradient operation is missing.
For custom distributed code, compare a small model against a non-parallel reference. Check forward outputs, loss, and gradients within tolerances appropriate for the numeric precision before scaling to a large run.
Common mistakes
The first mistake is assuming tensor parallelism already shards all activations. It shards particular model dimensions and computations; some intermediate tensors can remain replicated.
The second is treating sequence parallelism as independent data parallelism. Data-parallel workers normally process different samples and synchronize parameter gradients. Sequence-parallel workers cooperate on different token slices of the same logical activation inside a tensor-parallel computation.
The third is expecting sequence parallelism to solve every long-context bottleneck. It can reduce activation replication, but long sequences also increase attention computation and can create attention-specific memory pressure. Context parallelism, memory-efficient attention kernels, or other techniques may be needed when those costs dominate.
The fourth is combining parallelism modes without tracking the process groups. In a large training job, a worker can belong to data-, tensor-, pipeline-, and possibly context-parallel groups simultaneously. Collective operations must run over the intended group. A collective over the wrong workers can produce incorrect results or a distributed deadlock.
When to use sequence parallelism
Sequence parallelism is a strong candidate when all of these are true:
- training already uses tensor parallelism;
- replicated activations are a meaningful part of peak memory;
- longer sequences, larger micro-batches, or model size are pushing workers toward their memory limit;
- the training framework has a tested implementation for the model architecture and tensor-parallel configuration.
It is less compelling when a model fits comfortably, tensor parallelism is not otherwise needed, or memory is dominated by parameters and optimizer states rather than activations. In those cases, adding another distributed layout may increase operational complexity without addressing the real bottleneck.
If the problem is specifically attention over sequences too long for one worker, investigate context parallelism or an attention-specific strategy. If the job runs on one accelerator, activation checkpointing is a more directly applicable activation-memory technique.
Measure the result at the training-job level
A useful evaluation should compare the same model, data, effective batch size, precision, and optimizer with sequence parallelism disabled and enabled. Record at least:
- peak memory per worker;
- training step time or tokens processed per second;
- loss over a short controlled run;
- communication time if the profiler exposes it.
The loss check guards against distributed correctness problems. The memory measurement tells you whether the intended tensors were actually sharded. Throughput reveals whether the collective implementation and hardware topology make the trade worthwhile.
Avoid changing micro-batch size at the same time as the first comparison. A larger micro-batch may be the benefit you ultimately want, but changing it immediately makes it harder to isolate what sequence parallelism itself changed.
Conclusion
Sequence parallelism is best understood as an activation-layout optimization for tensor-parallel transformer training. Instead of keeping complete copies of token-wise activations on every tensor-parallel worker, it uses sequence shards between communication points and reconstructs the required layout when parallel computation demands it.
The practical payoff is lower activation memory per worker, which can make room for longer sequences or larger micro-batches. The boundary is equally important: it does not divide every part of training memory, remove attention’s cross-token dependencies, or replace every other memory technique.
When tensor parallelism is already necessary, profile which activations remain replicated. If those tensors are driving peak memory, sequence parallelism is a targeted way to remove that duplication without changing the model’s mathematical objective.