Positional Encoding

Understand Positional Information in Transformer Models

Self-attention can compare every token with other tokens in a context, but the comparison alone does not tell the model where those tokens occur. A sentence is not just a collection of words: changing their order can change the meaning. Transformer models therefore need a way to represent positional information. This mechanism lets the network distinguish, for example, the first occurrence of a token from a later occurrence and reason about relationships such as “the previous token” or “far earlier in the document.”