https://nalar.dev/tags/position-encoding/ Recent content in Position Encoding on Nalar Hugo en-us Mon, 14 Sep 2026 00:00:00 +0700 https://nalar.dev/encode-token-distance-with-rotary-position-embeddings/ Mon, 14 Sep 2026 00:00:00 +0700 https://nalar.dev/encode-token-distance-with-rotary-position-embeddings/ <p>Transformer attention has no intrinsic notion that one token sits three positions before another. Rotary position embeddings, usually called RoPE, inject position into attention by rotating pairs of query and key coordinates before their dot product is computed.</p> <p>The mechanism is easy to reduce to a helper function, yet several details determine its actual behavior: queries and keys must use compatible rotations, each coordinate pair has its own angular frequency, offsets emerge through the dot product, and changing the position scale changes the geometry seen by attention.</p>