Position Encoding

Encode Token Distance with Rotary Position Embeddings

Transformer attention has no intrinsic notion that one token sits three positions before another. Rotary position embeddings, usually called RoPE, inject position into attention by rotating pairs of query and key coordinates before their dot product is computed. The mechanism is easy to reduce to a helper function, yet several details determine its actual behavior: queries and keys must use compatible rotations, each coordinate pair has its own angular frequency, offsets emerge through the dot product, and changing the position scale changes the geometry seen by attention.