Reduce Repetition with Unlikelihood Training

An autoregressive language model is usually trained to increase the probability of the observed next token. That positive objective does not directly state which plausible but unwanted alternatives should receive less probability. When repetitive tokens or phrases remain locally probable, ordinary next-token training can leave generation with a strong route back into content that has already appeared.

Unlikelihood training adds a negative signal for selected candidates. Instead of only rewarding the target token, the objective can also penalize tokens chosen because they represent an unwanted behavior, such as repetition within the generated prefix.

The mechanism is simple, but candidate selection determines what behavior the penalty actually targets.

The standard objective rewards the observed token

For a token sequence x_1, ..., x_T, an autoregressive model assigns a conditional probability to each next token:

p(x_t | x_<t)

A standard token-level negative log-likelihood term is:

L_MLE(t) = -log p(x_t | x_<t)

Minimizing this term increases the probability of the observed target x_t under its prefix. Other token probabilities move indirectly because the output distribution is normalized, but the objective does not identify a specific undesirable token and state that it should be suppressed.

That distinction becomes relevant when a failure has a recognizable negative candidate set. If a token would create an unwanted repetition, it can be penalized explicitly rather than relying only on competition with the positive target.

Unlikelihood assigns loss to negative candidates

Let C_t be a set of tokens considered undesirable at position t. A token-level unlikelihood term can be written as:

L_UL(t) = -sum_{c in C_t} log(1 - p(c | x_<t))

For a candidate with low probability, 1 - p(c | x_<t) is close to one and the penalty is small. As the candidate probability approaches one, the penalty grows sharply.

The positive and negative terms can be combined, commonly with a weight controlling the contribution of the unlikelihood component:

L(t) = L_MLE(t) + alpha * L_UL(t)

Here alpha controls the relative strength of the negative objective. Its appropriate value depends on the candidate construction, data, model, and evaluation target. It is not a universal constant.

The formulation also makes one implementation constraint clear: the positive target should not be treated casually as a negative candidate at the same position. Conflicting objectives on the same token can make the intended update ambiguous.

Candidate construction defines the behavior being penalized

For repetition control, a candidate set can be derived from tokens that already occurred in the relevant context. If the current target is not itself a repetition that should be discouraged, previously seen tokens can form negative candidates.

A conceptual rule is:

C_t = tokens_seen_in_selected_prefix - allowed_tokens_at_t

The exact prefix and exclusion rule matter. Penalizing every previously seen token can be too broad for text in which legitimate repetition is common. Function words, entity names, code identifiers, structured fields, and deliberate lexical reuse can all recur for valid reasons.

Candidate construction can therefore use narrower signals, such as repeated n-grams or tokens associated with a specific degeneration pattern. A narrower candidate set reduces the number of unrelated probabilities receiving a direct negative update, but it also targets a narrower class of failures.

The loss function cannot decide which repetition is semantically acceptable. That decision enters through candidate generation and the data used to evaluate the result.

The penalty behaves differently from a generation-time repetition rule

A decoding rule can modify logits after the model has produced them. For example, a repetition penalty can reduce scores for tokens already present in the generated prefix. That intervention changes a particular decoding run without changing the model parameters.

Unlikelihood training acts earlier. It changes parameter updates so that selected negative candidates can receive lower probability under the contexts used during training.

The two mechanisms therefore operate at different layers of the system. A decoding penalty can be adjusted or removed at serving time. A model trained with an unlikelihood objective carries the effect in its parameters, subject to the contexts and candidate patterns represented during training.

This difference also affects evaluation. A decoding penalty should be evaluated together with the sampler or search procedure that applies it. An unlikelihood-trained model should be compared under controlled decoding settings so that parameter changes are not confused with a different inference rule.

Negative candidates compete through a normalized distribution

Language-model probabilities come from a normalized vocabulary distribution. Reducing probability on selected candidates necessarily redistributes probability mass elsewhere.

That redistribution is not guaranteed to move only to desirable alternatives. If the negative set is broad or poorly aligned with the task, the model can shift probability toward tokens that avoid the penalty without improving semantic quality.

This is one reason repetition metrics alone are insufficient. A system could reduce repeated n-grams while also becoming less coherent, less faithful to the prompt, or less accurate on the target task. The intended behavior has to be measured alongside the failure targeted by the negative objective.

The same point applies to alpha. Increasing the unlikelihood weight strengthens pressure against negative candidates, but a stronger penalty is not automatically a better model. The useful range is empirical and task-dependent.

Sequence-level candidates can target repeated continuations

Token-level candidates operate on individual next-token probabilities. Repetition can also be defined over longer patterns, such as an n-gram that has already appeared.

A sequence-level construction can identify a token as negative when choosing it would complete an unwanted repeated n-gram under the current prefix. The loss can still be applied to the next-token probability, but the candidate decision uses a longer context pattern.

For example, if the prefix has already contained:

vector search returns

and the current suffix is:

vector search

then returns can be marked as a negative candidate if completing that repeated trigram is the behavior being targeted.

This is more selective than penalizing every token seen earlier. It also means the behavior depends on the chosen n-gram order. Short patterns capture more repetitions but can include ordinary language reuse; longer patterns are more specific but miss shorter loops.

Candidate generation must match the model context used for the loss

Negative candidates are conditional on a prefix. If candidate generation examines information unavailable at the modeled position, the training signal no longer corresponds to the autoregressive context used by the probability term.

For repetition-based candidates, the relevant history is normally the prefix preceding the current token. The implementation should make that boundary explicit, especially in packed batches or shifted-label pipelines where tensor positions can obscure which tokens belong to the causal history.

Padding and packed examples also require care. Tokens from another example must not enter the candidate history merely because they share a physical tensor. Candidate construction should follow the same logical example boundaries as the causal objective.

This is separate from the attention mask. A correct mask can prevent cross-example attention while a preprocessing bug still builds negative candidates from the wrong segment.

Data distribution sets the meaning of repetition

Repeated text is not uniformly undesirable. Source code repeats identifiers. Dialogue repeats names and short acknowledgments. Technical documents reuse domain terms. Structured output can require exact field names many times.

A repetition candidate rule derived from one domain can therefore impose the wrong preference in another. The issue is not limited to domain vocabulary. Different output formats have different legitimate repetition structures.

Evaluation should preserve those distinctions. Alongside aggregate repetition rates, inspect task slices where repetition is required for correctness. For structured generation, exact schema adherence may matter more than lexical diversity. For summarization, preserving repeated proper nouns can be necessary for factual consistency.

A negative objective is most defensible when its candidate rule corresponds to a failure that can be stated precisely and measured separately from legitimate reuse.

Probability diagnostics can reveal overcorrection

Because unlikelihood training directly changes probabilities, evaluation can inspect more than final decoded text. For contexts with known negative candidates, compare their probability before and after adaptation. At the same time, track probability assigned to the correct target and to valid repeated tokens.

This separates two questions: whether the objective suppresses the candidates it was designed to suppress, and whether that suppression damages desirable alternatives.

Decoded outputs remain necessary because probability shifts can interact with sampling temperature, top-k filtering, nucleus sampling, or beam search. A moderate probability reduction may have little visible effect under one decoder and a large effect under another if it moves a token across a truncation boundary.

Unlikelihood training is therefore less a generic anti-repetition switch than a way to encode explicit negative candidates into the training objective. Its value depends on a precise definition of the unwanted behavior, candidate boundaries that match the autoregressive context, and evaluation that distinguishes reduced degeneration from reduced legitimate repetition.