Transformer attention has no intrinsic notion that one token sits three positions before another. Rotary position embeddings, usually called RoPE, inject position into attention by rotating pairs of query and key coordinates before their dot product is computed.
The mechanism is easy to reduce to a helper function, yet several details determine its actual behavior: queries and keys must use compatible rotations, each coordinate pair has its own angular frequency, offsets emerge through the dot product, and changing the position scale changes the geometry seen by attention.
Position acts as a rotation
Take one two-dimensional pair from a query vector. At position m, RoPE applies a rotation with angle m * theta:
R(m) = [
cos(m * theta) -sin(m * theta)
sin(m * theta) cos(m * theta)
]For an unrotated pair q, the positioned pair is:
q_m = R(m) qA key pair at position n receives the corresponding rotation:
k_n = R(n) kA full attention head contains many such pairs. RoPE assigns different angular frequencies to different pairs, so position is represented across multiple rotation rates rather than by one angle shared by the entire vector.
The operation preserves the Euclidean norm of each rotated pair because a rotation matrix is orthogonal. Position therefore changes orientation in these coordinate planes rather than increasing the vector magnitude merely because its index is larger.
Relative offsets appear inside the dot product
The useful property becomes visible when a rotated query is compared with a rotated key:
q_m^T k_n
= q^T R(m)^T R(n) k
= q^T R(n - m) kFor standard planar rotations, the product of the two position rotations reduces to a rotation based on the offset n - m. The attention score can therefore contain relative-position information even though each query and key was transformed using its own absolute index.
This does not mean attention depends only on distance. The original query and key contents still determine the dot product. RoPE changes the positional structure of that content-dependent comparison.
The sign of the offset is also retained. An offset of +4 corresponds to the inverse rotation of -4, so the transformation can distinguish tokens on opposite sides when the attention mask and architecture permit both directions.
Frequencies set several positional scales
If every coordinate pair rotated at the same rate, the representation would expose only one periodic positional pattern. RoPE instead uses a set of frequencies. A common construction derives an inverse frequency for pair index i from a base value and the head dimension.
The exact formula and pairing layout are architecture choices. A checkpoint expects the convention used when its parameters were trained. Changing the base, coordinate pairing, rotary dimension, or position indexing at inference time changes the query-key geometry even when all trained weights stay fixed.
Higher-frequency pairs rotate more rapidly as token positions increase. Lower-frequency pairs change more slowly. Together they provide attention with positional signals at different scales.
Periodicity still matters. Sine and cosine repeat, and the combined multi-frequency representation is not equivalent to an unbounded scalar distance. A model’s usable context is therefore not established merely by the fact that rotation values can be computed for arbitrarily large indices.
RoPE belongs on queries and keys
RoPE is commonly applied after query and key projections and before their attention dot product. In its standard form, it does not require adding a positional vector to the residual stream.
That placement has a practical consequence for cached autoregressive inference. Previously computed keys in a key-value cache already encode their positions. A new query must be rotated using its current sequence position, while newly appended keys must use matching position indices.
Reapplying a new position rotation to old cached keys would alter their meaning. Treating every newly decoded token as position zero would remove the intended offset relation between the new query and prior keys.
Position IDs also need to follow the model’s sequence semantics. Padding, packed sequences, prefix reuse, or cache slicing can make a tensor index differ from the logical token position. The rotation should use the position convention expected by the model, not an incidental memory offset.
Partial rotary dimensions change the scope
Some architectures rotate every query and key coordinate, while others reserve only part of each head for RoPE. If a head has width d and only r coordinates are rotary, the remaining d - r coordinates pass through without the position rotation.
The resulting attention score then combines a position-modulated component with an unrotated component. This is not interchangeable with full-head rotation, even if both configurations use the same frequency base.
The rotary width also has a structural constraint: the rotated coordinates must be arranged into pairs. Implementations may pair adjacent coordinates or use an equivalent split layout. Those layouts can produce the same mathematics when used consistently, but mixing conventions between training and inference does not preserve the transformation.
Context extension changes the position mapping
A model trained over one position range may be asked to process a longer range. RoPE can compute rotations beyond the training limit, but raw extrapolation exposes the model to angles and relative offsets outside the range represented during training.
Context-extension methods can alter position indices, rotation frequencies, or both. Such methods should be treated as changes to the positional mechanism rather than as a larger allocation for an otherwise identical model. Two systems using the same checkpoint but different RoPE scaling rules can produce different attention scores for the same long sequence.
This distinction is especially relevant when a runtime exposes a context-length setting separately from RoPE parameters. Increasing an allocation limit does not establish that the checkpoint was trained or adapted for the resulting positional range.
Rotation must stay consistent across execution paths
Optimized attention kernels often fuse position rotation with query-key preparation. A reference implementation may perform the same operation as separate tensor transforms. Numerical results should agree within the expected floating-point tolerance when both paths use the same position IDs, frequencies, rotary width, pairing convention, and precision behavior.
Cache reuse makes consistency more demanding. If a prefix cache was produced under one RoPE configuration, consuming it under a different position scale or frequency base mixes incompatible key representations. The cache is not just token content; its keys already contain positional rotation.
The same issue applies when moving checkpoints between runtimes. Parameter conversion can be correct while position handling is not. Comparing a few logits at several sequence positions is a more direct check than verifying tensor shapes alone.
RoPE is compact because relative offsets emerge from a structured query-key transformation rather than from a separate table of pairwise distances. That compactness also makes configuration details part of model semantics: position indices, frequencies, coordinate layout, and cache handling must describe the same rotation system from training through inference.