A transformer using rotary position embeddings can accept tensors longer than the sequence length used during training, yet accepting the shape does not establish that its position signal remains usable at those distances. Rotary angles at unseen positions can place attention computations outside the positional regime the model encountered during optimization.
Positional interpolation changes the input to rotary position embeddings rather than merely raising a sequence-length limit. For a target context longer than the original training context, position indices are compressed so the extended sequence maps into the earlier positional range. The model then needs to adapt to denser positional spacing instead of extrapolating directly to larger indices.
That distinction matters in implementations because context capacity is not only a tokenizer or buffer setting. The mapping from token index to rotary phase is part of model behavior.
RoPE turns position into rotation
Rotary position embeddings apply position-dependent rotations to query and key components. For one two-dimensional component pair, the rotation angle can be represented as
angle = position * frequencyDifferent component pairs use different frequencies. Attention between rotated queries and keys therefore contains information tied to relative position through their phase differences.
If a model was trained with positions from 0 through roughly L - 1, extending an input to a much larger index range asks the rotary mechanism and the surrounding model to operate on phase patterns that were not presented at the original context scale.
Changing an application limit from L to 4L does not alter that fact. It only permits larger indices to reach the model.
Interpolation compresses the position axis
For an original context scale L and a target scale L', with L' > L, linear positional interpolation uses a scale factor such as
scale = L / L'
mapped_position = position * scaleA token at target position L' is therefore mapped near position L on the rotary position axis. Intermediate target positions are compressed proportionally.
With a fourfold extension, the mapping is conceptually
target index: 0 1 2 3 4 ... 4L
mapped index: 0 0.25 0.50 0.75 1.00 ... LThe physical sequence still contains all target tokens. Only the positions supplied to the rotary calculation are rescaled.
This avoids direct use of rotary positions far beyond the original range, but it introduces a different shift: adjacent tokens now have smaller phase separation than they had under the original mapping. Fine-tuning can adapt model parameters to that compressed positional geometry.
Compression changes local as well as distant spacing
A common mental model treats context extension as a change affecting only tokens beyond the old limit. Linear interpolation is broader than that. Once the position axis is scaled, every nonzero positional distance inside the extended sequence is represented at the compressed scale.
For an extension factor of four, a physical separation of 400 tokens maps to a rotary separation corresponding to 100 positions under the original scale. A separation of four tokens maps to one original-scale position.
The model therefore does not receive the original positional representation for short distances while gaining a separate representation for long distances. The entire distance axis is compressed under basic linear interpolation.
This creates an implementation boundary: applying interpolation at inference to a model that was never adapted for the new mapping is not equivalent to using a model fine-tuned with that mapping. The position transformation may be identical, but the parameter state has seen different positional statistics.
The scale convention must match the implementation
Libraries expose RoPE scaling through different configuration fields and may define their factors from opposite directions. One interface may describe the extension ratio L' / L; another internal calculation may multiply positions by its reciprocal L / L'.
Those numbers describe related quantities but are not interchangeable without checking the convention.
For example, extending an 8,000-token positional range to 32,000 tokens gives
extension ratio = 32000 / 8000 = 4
position multiplier = 8000 / 32000 = 0.25Passing 4 into an API that expects a direct position multiplier would expand positions rather than compress them. Passing 0.25 into an API that expects an extension ratio can likewise produce the wrong transformation.
The reliable object to inspect is the effective rotary angle calculation. Configuration names are secondary to the mapping they produce.
Context length and usable context are separate properties
A model configuration can advertise a larger maximum position while still failing to use distant information reliably. Memory allocation, attention kernels, and cache indexing determine whether the sequence can execute. Positional adaptation determines a different part of the behavior.
Evaluation for an extended model therefore needs inputs that place relevant information across the new distance range. Testing only short prompts confirms that the modified model still handles short contexts; it does not establish that tokens near the new limit influence output as intended.
The reverse check also matters. Since interpolation changes positional spacing throughout the sequence, long-context adaptation can alter behavior on inputs that fit inside the original range. Short-context evaluation remains relevant even when the engineering goal is a larger window.
No single context-length number captures both execution support and effective use of distant tokens.
KV caches must use the same positional mapping
Autoregressive generation usually caches keys and values from earlier tokens. With RoPE, cached keys were produced using position-dependent rotations. A continuation must use positions consistent with the mapping applied when those cached entries were created.
Changing the RoPE scale in the middle of a cached sequence makes the old and new entries represent positions under different coordinate systems. The cache can remain structurally valid while its positional semantics are inconsistent.
This also affects cache reuse across requests or configurations. A cache produced under one rotary scaling policy should not be assumed compatible with another policy merely because model weights and tensor shapes match.
For systems that expose runtime context settings, the positional mapping belongs in the cache identity alongside the model and other state that changes key or value computation.
Interpolation does not remove attention cost
Positional interpolation addresses position representation. It does not by itself change the computational complexity of dense self-attention.
If sequence length grows by a factor of four, the number of query-key position pairs in a full attention matrix grows by a factor of sixteen. Optimized attention kernels can avoid materializing the complete matrix and can change practical memory behavior, but the amount of pairwise attention work still grows sharply with sequence length under dense attention.
KV-cache memory during autoregressive decoding also grows with the number of cached token positions unless another cache strategy changes that relationship.
A positional method can make a longer index range meaningful without making that range cheap to process. Context extension and context efficiency are separate design problems.
Positional interpolation is best treated as a coordinate transformation with model-level consequences. It keeps rotary positions within a familiar numerical range by compressing distances, then relies on adaptation to make that denser geometry useful. The implementation is sound only when training, inference, evaluation, and cache handling agree on the same position mapping.