A language model can assign high probability to a token that has already appeared several times in the generated text. If the decoder keeps selecting that token or a short pattern containing it, the output may settle into repetition even though each individual choice is plausible under the model.
A repetition penalty changes this behavior at decoding time. It modifies candidate scores according to token history before the next token is selected. The model parameters stay fixed, but the effective distribution used by the decoder no longer matches the model’s unmodified next-token distribution.
The penalty operates between model scoring and token selection
An autoregressive model produces a logit z_i for each vocabulary token i. Without an added decoding rule, those logits can be converted to probabilities with softmax and then consumed by greedy selection, sampling, or another search procedure.
A repetition-aware decoder inserts another transformation:
model logits -> repetition adjustment -> other decoding transforms -> token selectionThe adjustment depends on generated context. A token absent from the tracked history can retain its original score, while a token that has appeared can receive a lower effective preference. The exact transformation is part of the decoder implementation, not a universal property of language models.
This distinction prevents a common interpretation error. A penalized token is not assigned a different score by the neural network itself. The decoder is overriding part of the model ranking according to an external policy.
The ordering of decoding transforms can matter as well. Temperature scaling, token masking, truncation, and repetition adjustment are all operations on the candidate distribution or its scores. Two systems that use the same numeric penalty can still produce different candidate sets when they apply different transforms or define the penalty differently.
Token presence and token frequency encode different signals
A decoder can react to repetition using more than one statistic. A presence-based rule asks whether a token has occurred at least once. Once present, another occurrence receives the same adjustment regardless of whether the token appeared once or ten times.
A frequency-based rule instead uses the occurrence count. In a simple additive form, an adjusted logit can be written as:
z'_i = z_i - lambda * count_iHere count_i is the number of tracked occurrences and lambda is the penalty coefficient. A token used four times receives four times the subtraction applied to a token used once. This formula illustrates one policy; a specific inference system can use another transformation.
The two signals address different output patterns. Presence adjustment encourages vocabulary turnover after first use. Frequency adjustment becomes stronger as the same token accumulates. Combining them produces a score shift with both a fixed reuse cost and a count-dependent component.
Neither statistic directly measures semantic repetition. Token counts operate on tokenizer units. Repeating an idea with different tokens can avoid the penalty, while legitimate reuse of a common token can trigger it.
Multiplicative penalties require care around negative logits
Some decoders use a multiplicative-style repetition rule rather than subtracting a fixed amount. A naive operation such as dividing every repeated-token logit by a factor greater than one does not consistently reduce preference when logits can be negative.
For example:
original logit: -2.0
naive division by 1.2: -1.67The adjusted value is larger, so that operation raises the token’s relative score instead of lowering it. A sign-aware rule can avoid this reversal by transforming positive and negative logits differently. The precise rule must be treated as implementation-specific and checked in the inference stack being used.
This issue also shows that logits are not probabilities. Their absolute values can be shifted without changing the softmax distribution, and zero has no special probability meaning. A repetition transform defined directly on logits therefore needs to be evaluated by its effect on relative candidate scores, not by an intuition that a smaller magnitude necessarily means a stronger penalty.
Tokenization sets the granularity of repetition control
Repetition penalties usually track token identifiers rather than words. That makes tokenizer behavior part of the decoding policy.
Suppose a tokenizer represents related surface forms with different token sequences. Reusing the same visible word in another whitespace or punctuation context may not produce the same identifier sequence. Conversely, a frequent subword can occur across several unrelated words and accumulate a penalty even when the visible text is not repeating a phrase.
The history boundary matters too. A decoder might count only generated tokens, or it might include some prompt tokens in the tracked set. It can also restrict counting to a recent window rather than the full active context. These choices change which candidates are affected without changing the model output logits.
For applications that require exact phrase suppression, token-level repetition penalties are therefore an indirect control. Sequence constraints or explicit n-gram tracking can express that requirement more directly because they operate on ordered token patterns rather than independent token counts.
Strong penalties can damage necessary reuse
Repeated tokens are not inherently defective. Source code reuses identifiers and punctuation. Structured data repeats field names and delimiters. Technical prose may need the same term in adjacent sentences. A decoder that treats every reuse as evidence of degeneration can push probability mass toward less suitable alternatives.
The effect becomes more visible when the base distribution is already narrow. If one repeated token is strongly preferred by the model and the penalty demotes it below several weak alternatives, the decoder is no longer making a small correction. It has changed the local ranking that drives generation.
This can interact with sampling truncation. A penalized token may fall outside a retained candidate set after score adjustment. Once excluded, its original model probability no longer matters for that selection event. Applying a larger coefficient can therefore create discontinuous output changes when candidate-set membership changes.
For constrained formats, the same issue can affect validity. A repeated closing delimiter, property name, or syntax token may be required by the format. If a hard constraint system masks invalid candidates, that constraint should define the admissible set; repetition control should not be assumed to preserve structural validity on its own.
Repetition metrics need to match the failure mode
A single repetition rate can hide distinct behaviors. Consecutive duplicate tokens, recurring short n-grams, repeated sentences, and semantic restatement are different phenomena. A token-count penalty has the most direct connection to the first category and only an indirect connection to the others.
Evaluation should therefore preserve the level at which the decoder acts. Token reuse counts can reveal whether a penalty is changing its immediate target. N-gram recurrence can show whether short loops remain even when individual token frequencies look acceptable. Task-specific checks can then detect damage to required repetition, such as identifiers in code or keys in structured output.
The unmodified decoder is a useful reference point because the penalty deliberately changes the model distribution. Comparing both settings on the same prompts separates a genuine reduction in unwanted loops from a broader reduction in all repeated vocabulary.
A repetition penalty is most predictable when treated as a local decoding policy with explicit scope, counting rules, and score semantics. It can redirect token selection away from recurring candidates, but it does not identify repeated meaning and it does not repair a model distribution that assigns excessive mass to an undesirable sequence pattern. When the failure is sequence-level, the control mechanism often needs to operate at that level as well.