Extend RoPE Context with Positional Interpolation
A transformer using rotary position embeddings can accept tensors longer than the sequence length used during training, yet accepting the shape does not establish that its position signal remains usable at those distances. Rotary angles at unseen positions can place attention computations outside the positional regime the model encountered during optimization. Positional interpolation changes the input to rotary position embeddings rather than merely raising a sequence-length limit. For a target context longer than the original training context, position indices are compressed so the extended sequence maps into the earlier positional range. The model then needs to adapt to denser positional spacing instead of extrapolating directly to larger indices.