Language models sometimes repeat a phrase, return to the same point, or fall into a short loop even when the prompt asks for a concise answer. A common response is to increase randomness, but temperature changes the whole next-token distribution. That can reduce repetition while also making unrelated choices less predictable.

Token penalties provide a more targeted control. They adjust the scores of tokens that have already appeared, making some repeated tokens less likely before the decoder chooses the next token. This can be useful for open-ended generation, but it is not a general quality switch: repeated tokens are often exactly what correct text requires.

This article builds a practical mental model for repetition penalties, compares common penalty designs, and explains how to tune them without accidentally damaging names, code, structured output, or other text where repetition is meaningful.

Repetition is a decoding problem only some of the time

An autoregressive language model generates one token at a time. At each step it produces a logit, an unnormalized score, for every possible next token. The decoder transforms those logits into a probability distribution and selects or samples a token.

Suppose a model has just generated:

The cache stores recently computed values. The cache

Continuing with stores may be perfectly reasonable. In a different context, repeatedly producing:

The cache stores values. The cache stores values. The cache stores values.

is undesirable.

The important point is that the decoder sees token probabilities, not a semantic label saying “this repetition is useful” or “this repetition is a loop.” A token penalty therefore uses a simpler signal: whether a token occurred before, or how often it occurred.

That makes the mechanism easy to apply, but also explains its main limitation. It discourages lexical reuse, not repetition of meaning.

Think of a penalty as editing logits before sampling

Let z_i be the model’s logit for token i. Without a repetition control, a simplified decoding pipeline is:

model -> logits -> temperature / filtering -> sample next token

A token penalty inserts a history-dependent adjustment:

model -> logits -> history penalty -> temperature / filtering -> sample next token

The exact order and formula depend on the inference implementation. The reusable idea is that generation history changes the score used for a token before selection.

Consider a tiny vocabulary with these illustrative logits:

Token       Original logit
cache             4.2
stores            3.8
keeps             3.4
memory            2.9

If cache has already appeared several times, a penalty might reduce its effective score while leaving unseen tokens unchanged:

Token       Original     Adjusted
cache          4.2          3.1
stores         3.8          3.8
keeps          3.4          3.4
memory         2.9          2.9

The model has not been retrained. The decoder has only changed the distribution from which the next token will be chosen.

Presence and frequency penalties answer different questions

Two useful penalty designs differ in what they measure.

A presence penalty asks whether a token has appeared at least once. A simple additive form is:

adjusted_logit(i) = logit(i) - alpha * seen(i)

where seen(i) is 1 if token i has appeared in the relevant history and 0 otherwise.

Once a token has appeared, its penalty does not grow just because it appears again. This tends to encourage the decoder to move toward tokens it has not used yet.

A frequency penalty instead depends on the number of previous occurrences:

adjusted_logit(i) = logit(i) - beta * count(i)

If a token has appeared four times, it receives four times the adjustment that a token seen once receives in this simplified design. That makes frequency-based penalties more directly aimed at repeated reuse.

For example, suppose cache has occurred three times and memory once, with beta = 0.4:

Token       Original     Count     Adjusted
cache          4.2         3          3.0
memory         2.9         1          2.5
stores         3.8         0          3.8

These equations are teaching models, not universal API definitions. Providers and inference libraries can expose different parameter names, ranges, formulas, scopes, and processing order. Treat the documentation for the runtime you use as the contract.

A repetition penalty can use a different transformation

Some decoders expose a parameter named simply repetition penalty rather than separate presence and frequency controls. Do not assume that this is another name for the additive formulas above.

One implementation can rescale logits for previously generated tokens instead of subtracting a fixed amount. Another can inspect a limited recent window rather than the entire generated sequence. Tokenization and the order of other logit processors can also affect the result.

The safe mental model is therefore:

presence penalty  -> depends on whether a token appeared
frequency penalty -> depends on how often a token appeared
repetition penalty -> implementation-specific history-based logit transform

When moving a generation configuration between APIs or inference engines, matching parameter values does not guarantee matching behavior. Verify the actual definition before treating two settings as equivalent.

Tokenization makes penalties less intuitive than word counting

Penalties usually operate on tokens, not words. A word can consist of one token in one context and several tokens in another, depending on the tokenizer.

Imagine that a tokenizer represents a product identifier as several pieces:

ZX-2048 -> [ZX] [-] [20] [48]

If those pieces recur elsewhere, a token-level penalty can affect the identifier even though the decoder is not explicitly counting repeated product names. Common punctuation, whitespace-related tokens, and fragments inside longer words can also be affected.

This matters when diagnosing output. Counting repeated words in the rendered text is not enough to reproduce what the decoder saw. For precise debugging, inspect the actual token IDs and the history scope used by the penalty implementation.

Penalties and temperature solve different problems

Temperature changes the relative sharpness of the probability distribution across many tokens. A higher temperature generally makes lower-scoring alternatives more competitive; a lower temperature concentrates probability more strongly on high-scoring choices.

A repetition penalty is conditional on generation history. It can reduce the score of a repeated token while leaving an unseen token’s logit untouched.

That difference suggests a useful tuning rule: if the output is otherwise coherent but repeats specific tokens or phrases too aggressively, test a small repetition-oriented adjustment before changing global randomness. If the broader problem is that generation is too deterministic or too variable, temperature may be the more relevant control.

The controls still interact. After penalties modify logits, temperature or probability filtering can amplify or reduce the practical effect on the final sampling distribution. Tune the complete decoding configuration rather than assuming each parameter acts independently.

Start with the weakest intervention that fixes the failure

A practical evaluation can use a small set of prompts that reliably expose the repetition problem. Keep the prompt, model, maximum output length, and other decoding settings fixed while varying one penalty at a time.

For each configuration, inspect more than repetition rate. Useful checks include:

  • whether repeated phrases or loops decrease;
  • whether required terms disappear;
  • whether factual statements remain coherent;
  • whether names and identifiers stay intact;
  • whether lists and structured fields remain complete;
  • whether output becomes unnaturally synonym-heavy.

If a small penalty fixes the observed loop, increasing it further usually adds risk without solving a new problem. The target is not minimum repetition. The target is appropriate repetition for the task.

For production evaluation, use representative inputs and task-specific quality checks rather than tuning from one attractive sample. Stochastic decoding also means that a single run can be misleading; compare enough generations to distinguish a consistent change from sampling variation.

Strong penalties can damage correct output

Natural language contains necessary repetition. Articles, prepositions, technical terms, variable names, and entity names often need to recur. Code and structured data are even less tolerant of arbitrary lexical avoidance.

Consider a response that must explain a request_id field in several steps. Penalizing tokens that compose request_id may push the model toward inconsistent alternatives just when exact terminology is desirable.

The same problem appears in tasks such as:

  • generating code where identifiers must be reused;
  • producing JSON with repeated property names across objects;
  • quoting a fixed phrase accurately;
  • writing about a technical concept whose standard name should remain stable;
  • generating repetitive-by-design formats such as tables or templates.

A high penalty can also produce awkward paraphrasing. The decoder may avoid the most natural token because it appeared earlier, choosing a less suitable synonym even though the underlying idea is unchanged.

This is why lexical diversity is not the same as semantic quality.

Token penalties do not guarantee that loops disappear

A model can repeat an idea using different tokens:

The operation is inexpensive.
The procedure has low cost.
Running it does not require many resources.

A token-level penalty may see substantial novelty even though the answer is circling the same point.

Conversely, a generation loop can involve enough different tokens that a modest token penalty does not break it. Repetition may also originate from the prompt, retrieved context, fine-tuning data, or a task format that strongly encourages a repeated pattern.

If the failure is semantic rather than lexical, other interventions may be more appropriate: improve the prompt, remove duplicated context, set a clearer stopping condition, constrain the output format, change the decoding strategy, or evaluate whether the model itself is suitable for the task.

Penalties should address a measured decoding failure, not hide an upstream problem.

Decide the history scope deliberately

A penalty needs a definition of “already appeared.” Some systems consider all generated tokens; others can be configured around a recent window or another scope.

A full-history penalty can discourage a term used hundreds of tokens earlier even when repeating it now would be natural. A recent-window approach focuses more directly on local loops, but it allows older tokens to become unpenalized again.

Neither scope is universally preferable. For a short marketing variation, encouraging novelty across the entire response may be useful. For a long technical explanation, penalizing every earlier occurrence can fight necessary terminology. The appropriate scope follows from the kind of repetition you are trying to prevent.

Also check whether the implementation counts prompt tokens, generated tokens, or both. That detail can materially change behavior when the prompt contains terms that the answer must repeat.

Know when a simpler control is better

Use a token penalty when you have evidence that generation is overusing lexical material and the task permits wording variation. It is especially reasonable for open-ended text where mild diversity is desirable and exact token reuse is not part of correctness.

Prefer a simpler or more direct mechanism when the requirement is different. If an answer should stop after a known delimiter, use an appropriate stopping mechanism. If output must follow a schema, use structured or constrained decoding when the runtime supports it. If one exact phrase must never appear, a targeted token or string constraint is conceptually closer to the requirement than globally penalizing every repeated token.

For deterministic code, identifiers, or machine-readable structures, start without repetition penalties unless evaluation shows a specific benefit. Correct repetition is often part of the format.

Conclusion

Token penalties are history-dependent edits to an LLM’s next-token scores. Presence-style penalties react to whether a token has appeared, frequency-style penalties grow with repeated occurrences, and parameters called repetition penalties may use different implementation-specific transformations.

Their strength is targeted control: they can reduce lexical loops without changing every token score in the same way that global randomness controls do. Their weakness comes from the same mechanism. A decoder cannot infer from token counts whether repetition is semantically redundant or required for correctness.

Use penalties only after identifying a real repetition failure, verify the exact behavior of your inference runtime, and tune against representative outputs. The useful setting is the smallest one that reduces unwanted repetition while preserving the repetition the task actually needs.

Artificial Intelligence LLM Inference Decoding