https://nalar.dev/tags/language-models/ Recent content in Language Models on Nalar Hugo en-us Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-tokenization-in-language-models/ Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-tokenization-in-language-models/ <p>Language models do not read text as a sequence of words. Before a model can process a prompt, a <strong>tokenizer</strong> converts the text into a sequence of token IDs from a fixed vocabulary. The model operates on those IDs, and generated IDs are later converted back into text.</p> <p>This extra layer is easy to ignore because most model APIs accept ordinary strings. But tokenization affects how much text fits in a context window, how usage-based costs are calculated, how text is truncated or split, and why seemingly small formatting changes can alter model behavior.</p>