Reduce Repetitive Generation with Unlikelihood Training

A language model can learn to predict ordinary text well and still assign too much probability to behavior you don’t want at generation time. Repetition is a common example: once a phrase appears, the model may keep making recently used tokens plausible enough that a decoding loop becomes hard to escape.

Changing the decoder can hide some of this behavior, but it doesn’t change the probabilities learned by the model. Unlikelihood training takes a different approach. During training, it identifies undesirable candidates and explicitly pushes their probabilities down while the usual likelihood objective pushes desired tokens up.

This article builds the method from a small repetition example, explains the loss, and shows how negative-candidate design determines what the model actually learns. By the end, you’ll have a practical mental model for deciding when unlikelihood training fits a generation problem and what to validate before using it.

Likelihood training says what to prefer, not what to avoid

For an autoregressive language model, standard maximum-likelihood training predicts the next token from the preceding context. If the training sequence is:

The service restarted successfully

then at one step the model sees a prefix such as:

The service restarted

and is trained to increase the probability of the observed next token, successfully.

For target token y_t and context x_<t, the usual token-level negative log-likelihood is:

L_MLE = -log p(y_t | x_<t)

Minimizing this loss raises the probability assigned to the observed target. Other tokens compete indirectly because the model’s output probabilities must sum to one, but the loss doesn’t say which wrong tokens are especially undesirable.

That distinction matters when a failure has structure. Suppose a model generating a support response has already written:

Restart the service and check the logs. Restart the service

At the next position, another and may continue a repeated phrase. Standard likelihood training has no special instruction that says, “given this prefix, penalize the token that continues the repetition.” Unlikelihood training can provide that signal.

The mental model: give selected mistakes their own loss

At each training position, define a set of negative candidates: tokens that should be unlikely in the current context. Call that set C_t.

For a negative candidate c, a simple unlikelihood term is:

L_UL(c) = -log(1 - p(c | x_<t))

This is the mirror image of the usual likelihood idea. Likelihood penalizes the model when the desired token has low probability. Unlikelihood penalizes it when an unwanted token has high probability.

For several negative candidates, the token-level loss can be written as:

L_UL = - sum_{c in C_t} log(1 - p(c | x_<t))

A training objective can then combine the normal language-model loss and the unlikelihood term:

L = L_MLE + alpha * L_UL

Here, alpha controls the relative strength of the negative signal. This coefficient isn’t a universal constant. Its useful range depends on how candidates are selected, how many are selected, the model, and the task.

The shape of -log(1-p) is useful for this purpose. If an unwanted token already has tiny probability, its penalty is small. As its probability approaches 1, the penalty grows sharply. Training therefore concentrates more pressure on negative candidates the model currently considers very plausible.

A minimal repetition example

Consider this simplified token sequence:

red blue red

At the fourth position, imagine the desired next token is green. We decide that repeating a previously used content token is undesirable for this toy task, so the negative set is:

C_t = {red, blue}

Suppose the model predicts:

p(green) = 0.50
p(red)   = 0.30
p(blue)  = 0.10
other    = 0.10

The likelihood term pushes green upward. The unlikelihood term separately pushes red and blue downward:

L_MLE = -log(0.50)

L_UL = -log(1 - 0.30)
       -log(1 - 0.10)

Numerically, using natural logarithms:

L_MLE ≈ 0.693
L_UL  ≈ 0.357 + 0.105
      ≈ 0.462

If alpha = 0.5, the combined loss for this teaching example is approximately:

L ≈ 0.693 + 0.5 * 0.462
  ≈ 0.924

The exact number isn’t the main point. The two parts of the objective carry different information. L_MLE says which token should win. L_UL says which plausible alternatives deserve explicit downward pressure.

A production repetition rule would usually be more selective than “penalize every previous token.” Repeating words such as articles, punctuation, names, or necessary technical terms can be completely valid. Candidate construction is where the method becomes task-specific.

Negative candidates define the behavior you are training against

Unlikelihood training doesn’t discover undesirable behavior on its own. You have to define or obtain the negative candidates. A mathematically correct loss paired with poor candidates can train exactly the wrong preference.

For repetition control, a candidate rule might inspect a recent window and mark tokens that would recreate an unwanted n-gram. For a constrained dialogue task, negatives might come from known contradictory responses or another task-specific detector. The common pattern is the same:

context -> identify undesirable candidate -> penalize its probability

The negative set must also be compatible with the positive target. If the ground-truth next token is placed in C_t, the objectives conflict: likelihood tries to raise its probability while unlikelihood tries to lower it. That may happen accidentally when a broad heuristic labels legitimate repetition as a mistake.

This is why candidate precision often matters more than candidate quantity. Adding more negatives isn’t automatically stronger supervision. It can simply create more opportunities to suppress valid language.

Static negatives and model-generated negatives solve different problems

Some negatives can be derived directly from the reference sequence or task rules. These are cheap and predictable, but they only cover mistakes your rule can anticipate.

Another option is to generate outputs from the current model, detect undesirable behavior in those outputs, and use it to construct negative training signals. This exposes training to errors the model actually makes. It also adds generation cost and creates a moving data distribution because the model’s mistakes change as training progresses.

Neither approach is inherently superior. Rule-derived negatives are attractive when the failure is crisp and local. Model-generated negatives become more useful when the problem appears mainly in free-running generation rather than in teacher-forced training contexts.

Why decoding fixes and unlikelihood training are not interchangeable

A repetition penalty, n-gram blocking rule, or sampling strategy operates at inference time. It modifies which continuation is selected from model scores, or modifies the scores themselves, without retraining the underlying model.

Unlikelihood training changes model parameters. The goal is for the unwanted continuation to receive less probability before a decoder applies any additional constraints.

That difference creates a practical trade-off. A decoding rule is often the simpler first response when a failure is easy to detect, the model can’t be retrained, or the constraint must be exact. You can deploy it quickly and change it without another training run.

Training becomes more attractive when the undesirable behavior is persistent across decoding settings, when inference-time rules are too blunt, or when you want the model’s probability distribution itself to reflect the preference. It also costs more: you need suitable training data, a candidate-generation procedure, optimization, and evaluation for regressions.

The two approaches can also be combined. Training can reduce a tendency while decoding enforces a hard product requirement. Don’t assume training alone provides a guarantee. A lower learned probability is still a probability, and an autoregressive model can encounter contexts that weren’t represented by the training negatives.

Tune the pressure without erasing useful probability mass

The coefficient alpha, candidate-set size, and candidate frequency all affect the strength of unlikelihood training. Looking only at alpha can therefore be misleading.

Suppose configuration A selects one negative candidate at 10% of positions, while configuration B selects twenty negatives at most positions. Using the same alpha doesn’t make their optimization pressure comparable. The second setup contributes many more unlikelihood terms.

Monitor both the target behavior and ordinary language-model quality. For repetition, that means measuring repetition on generated continuations, not merely checking the training loss. At the same time, evaluate whether useful generations became less fluent, less accurate, or less willing to repeat information when repetition is required.

The evaluation set should contain counterexamples to your negative rule. If your system writes code explanations, for example, repeating an identifier can be necessary. If it summarizes incident reports, a service name may legitimately appear several times. A repetition metric that rewards lexical variety without checking meaning can make a damaged model look improved.

Common failure modes

The most common mistake is treating negative-candidate selection as a minor implementation detail. It is part of the learning objective. A heuristic that labels valid tokens as undesirable creates noisy or contradictory supervision.

Another mistake is evaluating only under one decoder. If you train specifically to alter the model distribution, inspect that distribution’s effects under the decoding methods you expect to deploy. A change that looks helpful under greedy decoding may interact differently with sampling or beam search.

Be careful with rare but required repetition as well. Consider:

The environment variable DATABASE_URL must match DATABASE_URL in the deployment configuration.

A crude rule against repeated tokens would penalize exactly the behavior needed for correctness. Similar problems occur with names, quotations, mathematical notation, structured formats, and code.

Finally, don’t interpret unlikelihood training as a general truthfulness or safety mechanism. The loss can push down candidates you can identify, but it doesn’t prove that every unpenalized continuation is correct or desirable. Broad properties such as factuality require evaluation and supervision designed for those properties.

When unlikelihood training is a good fit

Unlikelihood training is most compelling when three conditions line up: the failure is visible in model probabilities or generations, undesirable candidates can be identified with reasonable precision, and retraining or fine-tuning is available.

For a local and deterministic constraint, use the simpler tool first. If an application must never emit a particular token sequence, a hard decoding constraint can provide behavior that a learned penalty cannot guarantee. If you only have API access to a hosted model, training-time unlikelihood may not be available at all.

When you do control training, start with the narrowest negative rule that captures the failure. Compare against the same model without the unlikelihood term, generate outputs under the deployment decoder, and inspect examples where the rule fires. Then broaden the negative set only when evaluation shows that the added coverage helps more than it suppresses legitimate text.

Make the negative signal earn its complexity

The useful idea behind unlikelihood training is simple: positive examples aren’t the only way to shape a model distribution. When you can name a specific wrong continuation, you can give that mistake direct optimization pressure instead of hoping it becomes unlikely as a side effect of likelihood training.

The difficult part isn’t the formula. It’s deciding what deserves to be unlikely. Treat negative-candidate construction as supervision, test it against legitimate edge cases, and compare it with cheaper inference-time controls. If the negative signal is precise and the failure genuinely lives in the learned distribution, unlikelihood training gives you a focused way to teach the model what not to predict.