https://nalar.dev/tags/decoding/ Recent content in Decoding on Nalar Hugo en-us Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/control-llm-repetition-with-token-penalties/ Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/control-llm-repetition-with-token-penalties/ <p>Language models sometimes repeat a phrase, return to the same point, or fall into a short loop even when the prompt asks for a concise answer. A common response is to increase randomness, but temperature changes the whole next-token distribution. That can reduce repetition while also making unrelated choices less predictable.</p> <p><strong>Token penalties</strong> provide a more targeted control. They adjust the scores of tokens that have already appeared, making some repeated tokens less likely before the decoder chooses the next token. This can be useful for open-ended generation, but it is not a general quality switch: repeated tokens are often exactly what correct text requires.</p> https://nalar.dev/steer-llm-decoding-with-token-and-sequence-biases/ Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/steer-llm-decoding-with-token-and-sequence-biases/ <p>Sometimes an LLM produces generally good text but makes one narrow decoding choice too often. Perhaps a domain-specific abbreviation should be preferred, a deprecated product name should be discouraged, or a particular token must not appear in generated text.</p> <p>Changing temperature is a poor fit for this problem because temperature affects the whole next-token distribution. Retraining a model is usually excessive when the desired change is local. <strong>Token and sequence biases</strong> provide a narrower tool: modify selected prediction scores during decoding while leaving the model parameters unchanged.</p>