Language Models

Understand Tokenization in Language Models

Language models do not read text as a sequence of words. Before a model can process a prompt, a tokenizer converts the text into a sequence of token IDs from a fixed vocabulary. The model operates on those IDs, and generated IDs are later converted back into text. This extra layer is easy to ignore because most model APIs accept ordinary strings. But tokenization affects how much text fits in a context window, how usage-based costs are calculated, how text is truncated or split, and why seemingly small formatting changes can alter model behavior.