Transformers

Understand Tokenization in Large Language Models

Large language models do not read text as words or characters in the way people do. Before text reaches the model, a tokenizer converts it into a sequence of discrete units called tokens and maps those tokens to numerical identifiers. Tokenization is easy to overlook because most model APIs perform it automatically. Yet token boundaries affect context-window usage, inference cost, truncation, multilingual behavior, and even whether two visually similar strings are represented in similar ways. Understanding this layer makes many LLM behaviors easier to reason about.