Rotary Position Embedding (RoPE) applies position-dependent rotations to query and key coordinates before their attention dot product. The resulting score carries relative position through the phase difference between those rotations rather than through an additive position vector attached to the token representation.

This distinction is structural. RoPE starts from absolute indices for each rotation, yet the query-key inner product can be written in terms of the offset between their positions.

Paired coordinates carry independent angular frequencies

RoPE partitions an even-dimensional query or key vector into two-dimensional coordinate pairs. For one pair, position m applies a rotation

R(m, theta) =
[ cos(m theta)  -sin(m theta) ]
[ sin(m theta)   cos(m theta) ]

where theta is the angular frequency assigned to that pair. A full RoPE transform is block diagonal: each pair receives the same position index but generally a different frequency.

If q is a query and k is a key, the rotated vectors at positions m and n are

q_m = R(m) q
k_n = R(n) k

with R representing the block-diagonal collection of rotations.

The operation preserves the Euclidean norm of each rotated pair because each block is orthogonal. Position changes orientation, not vector length, inside the rotary subspace.

The dot product reduces absolute phases to an offset

The attention score contains

q_m^T k_n
= (R(m) q)^T (R(n) k)
= q^T R(m)^T R(n) k

For planar rotations,

R(m)^T R(n) = R(n - m)

so the positional term inside the dot product depends on n - m. The absolute indices still determine the two rotations that are executed, but their interaction in the inner product has a relative-position form.

This algebra is the central boundary of RoPE. It does not mean every hidden state in the model becomes translation invariant. Values, residual streams, nonlinear layers, masking, finite context boundaries, and other model components retain their own behavior. The relative-offset property applies to the rotary query-key interaction.

Multiple frequencies give one offset several phases

A single angular frequency is periodic. If only one pair existed, offsets separated by its full period would produce the same rotation for that pair. RoPE instead uses a bank of frequencies across coordinate pairs.

For pair i, the relative rotation is determined by

(n - m) * theta_i

and the full attention score combines contributions from all pairs. The same token offset is therefore represented through several phases at different angular rates.

The frequency schedule is part of the positional geometry. Changing the base, rescaling positions, or modifying selected frequencies changes the phases presented to attention. Such changes are not equivalent to merely increasing an implementation’s maximum sequence-length constant.

RoPE modifies queries and keys, not values by requirement

In the standard rotary construction, the positional rotation is applied to queries and keys because their inner product determines attention compatibility. The value vector does not need the same rotation to obtain the relative-offset identity above.

That separation matters for cache design during autoregressive inference. A cached key must correspond to the positional convention expected by later query-key products. Implementations can store keys before or after rotary transformation, but the cache and attention kernel must agree on that representation. A mismatch changes scores even when token content and model weights are identical.

The architectural property does not prescribe one cache layout. Pre-rotated caching and deferred rotation are implementation choices with different data-flow consequences.

Causal masking is separate from rotary position

RoPE changes the score associated with a query-key pair. It does not itself prohibit a token from attending to future positions.

In a causal decoder, a mask independently removes disallowed query-key pairs before attention probabilities are formed. The rotary phase can encode the offset for a pair while the causal mask determines whether that pair participates at all.

Keeping these mechanisms separate avoids assigning access-control semantics to position encoding. RoPE supplies positional structure to compatibility scores; the attention mask supplies the permitted connectivity pattern.

Longer indices remain computable without becoming guaranteed context

The rotation formula can be evaluated for position indices beyond those observed during training. Sinusoids do not require a finite lookup table with one stored vector per supported index.

Computability, however, is not a guarantee of stable model behavior at arbitrary lengths. At larger offsets, every frequency advances to phases that may occupy positional regimes not represented adequately during training. The model’s use of those phases depends on its trained parameters and data distribution.

Context-extension methods that alter RoPE frequencies or position scales target this boundary. They change the mapping from token offsets to angular phase so that a longer index range occupies a different region of the rotary spectrum. Their effects depend on the exact scaling rule, model, and adaptation procedure; the basic RoPE identity alone does not establish a universal long-context quality result.

RoPE’s precise contribution is narrower and more useful: it places position inside query-key geometry so that absolute rotations combine into relative phase offsets in the attention dot product. The frequency bank determines how those offsets are represented, while masking, cache representation, training range, and the rest of the transformer determine separate parts of system behavior.

Artificial Intelligence Transformers Attention Position Encoding RoPE