Position Interpolation Compresses RoPE Indices into the Original Context Range

Thu, 24 Sep 2026 00:00:00 +0000

A RoPE-based Transformer associates token positions with rotations whose angles depend on the position index and per-dimension frequencies. Feeding a sequence beyond the context range used during training pushes those rotations to position indices the model did not encounter in that regime. Position Interpolation changes that boundary by scaling the extended indices back into the original range before the rotary transformation is applied.

For an original context limit L and a target context L' > L, a simplified linear mapping is:

Transformer Inference on Nalar

Position Interpolation Compresses RoPE Indices into the Original Context Range