https://nalar.dev/tags/transformers/ Recent content in Transformers on Nalar Hugo en-us Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-tokenization-in-large-language-models/ Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-tokenization-in-large-language-models/ <p>Large language models do not read text as words or characters in the way people do. Before text reaches the model, a tokenizer converts it into a sequence of discrete units called tokens and maps those tokens to numerical identifiers.</p> <p>Tokenization is easy to overlook because most model APIs perform it automatically. Yet token boundaries affect context-window usage, inference cost, truncation, multilingual behavior, and even whether two visually similar strings are represented in similar ways. Understanding this layer makes many LLM behaviors easier to reason about.</p>