https://nalar.dev/tags/context-window/ Recent content in Context Window on Nalar Hugo en-us Sat, 12 Sep 2026 00:00:00 +0700 https://nalar.dev/extend-rope-context-windows-with-position-interpolation/ Sat, 12 Sep 2026 00:00:00 +0700 https://nalar.dev/extend-rope-context-windows-with-position-interpolation/ <h1 id="extend-rope-context-windows-with-position-interpolation">Extend RoPE Context Windows with Position Interpolation</h1> <p>A RoPE-based language model trained on sequences up to a fixed length can behave poorly when inference suddenly asks it to process much larger position indices. The tokens are valid, but the positional pattern can move far outside the range used during training.</p> <p><strong>Position interpolation</strong> changes that geometry. Instead of sending larger position indices directly into rotary position embeddings, it compresses a longer sequence into the positional range the model already uses. With suitable adaptation, this can extend the usable context window without changing the transformer architecture.</p>