A language model can produce fluent text and still get stuck repeating a phrase, sentence pattern, or idea. A support reply may restate the same apology three times. A summarizer may loop over one point. A long generation can begin copying a short phrase again and again.

The tempting fix is to turn up a “repetition” setting until the duplicate text disappears. That can solve one symptom while creating another: names become awkward, code becomes invalid, required terms disappear, or a model avoids repeating words that the task genuinely needs.

Repetition control is therefore a decoding problem, not simply a duplicate-removal switch. This article builds a practical mental model for it, compares soft penalties with hard n-gram constraints, and shows how to evaluate whether a control reduces harmful repetition without damaging the output.

First separate useful repetition from degeneration

Repeated tokens are not automatically a model failure. Consider this short answer:

Use HTTPS for the public endpoint. Keep HTTP only for the local health check.

HTTP appears inside HTTPS at the character level, and related protocol terms recur because the topic requires them. Source code, product names, legal text, tables, and structured output can require even more exact reuse.

The problem is unwanted repetition: generation keeps returning to content that adds little or no new information. For example:

Retry the request after 30 seconds.
If it still fails, retry the request after 30 seconds.
Then retry the request after 30 seconds.

A useful repetition policy must distinguish these cases indirectly. Most decoding controls do not understand whether repetition is semantically justified. They operate on token history and scores, so aggressive settings can suppress both bad loops and legitimate reuse.

That limitation should shape the whole design: use the weakest intervention that fixes the measured failure mode.

The mental model: modify the next-token decision

An autoregressive language model repeatedly predicts a distribution over the next token. Conceptually:

prompt + generated tokens
        |
        v
      model
        |
        v
next-token scores
        |
        v
generation rules
        |
        v
choose next token

Repetition controls usually act in the generation rules stage. They do not retrain the model or erase anything from its context. Instead, they alter which next tokens are attractive or allowed based on what has already been generated.

This distinction matters. If a model strongly prefers a repetitive continuation, decoding can steer it elsewhere, but the underlying model preference still exists. A decoding rule can also force a locally different token without making the resulting sentence globally better.

Two broad approaches are especially useful to understand:

  • soft controls reduce the attractiveness of repeated tokens but still permit them;
  • hard constraints make particular repeated patterns unavailable.

They solve related problems with different risks.

Start with a soft repetition penalty

Suppose a simplified decoder is choosing the next token after:

The cache is full, so clear the

Imagine the model assigns these illustrative scores:

cache     4.2
entry     3.9
buffer    3.4

If cache has already appeared, a repetition penalty can reduce its effective score before the decoder chooses or samples the next token. The exact transformation is implementation-specific; different libraries and APIs can define a parameter named “repetition penalty” differently.

The important concept is simpler than any one formula:

model score
    + information about previously generated tokens
    -> adjusted score

Because the control is soft, cache can remain possible when the model has strong reason to use it. That makes a soft penalty useful when repetition is undesirable but sometimes necessary.

Why a token-level penalty is imperfect

A token-level penalty does not directly represent repeated ideas or phrases. Tokenization can split words into multiple pieces, and the same idea can be expressed with different tokens. Conversely, a required identifier may need to appear many times even though every occurrence is valid.

Consider generated code:

if user_id:
    log(user_id)
    return load(user_id)

Penalizing user_id merely because it appeared earlier can work against the task. The decoder sees repetition; the program sees a necessary variable reference.

This is why repetition penalties should be tuned against real task outputs rather than chosen from a generic recommendation.

Hard n-gram constraints target exact local loops

A different strategy is to forbid the decoder from creating an n-gram that has already appeared in the generated sequence. An n-gram is a contiguous sequence of n tokens.

For a small teaching example, treat each word as one token and set n = 3. Suppose the output already contains:

restart the service

Later, the current suffix is:

... restart the

Choosing service would recreate the earlier 3-gram restart the service. A no-repeat 3-gram rule therefore blocks service at this step.

Conceptually, the decoder asks:

Would candidate token X complete an n-gram seen before?

no  -> X remains eligible
yes -> block X

Unlike a soft penalty, this is a hard rule. Once a candidate would violate the constraint, a stronger model preference cannot override it.

Why n matters

Small n-grams are common in normal language. Banning repeated unigrams would prevent reuse of any token and is unsuitable for ordinary prose. Banning repeated bigrams can also be restrictive because phrases such as of the or in the naturally recur.

Larger n-grams target more specific repeated spans and therefore interfere less with ordinary word reuse. But they also miss loops that vary slightly each time.

There is no universally correct n. The useful value depends on tokenization, output length, language, and task structure. Treat it as an evaluated generation parameter, not a property of the model.

Soft penalties and hard constraints behave differently

Suppose a generated answer legitimately needs to say API key in two separate sections.

A soft token penalty may make API and key somewhat less attractive the second time, but the model can still select them if their adjusted scores remain competitive. A hard no-repeat 2-gram constraint would forbid the exact token pair if it has already occurred.

That difference leads to a practical rule of thumb:

soft penalty   -> discourage recurrence
hard constraint -> prohibit a specific recurrence pattern

Use a hard rule when the prohibited pattern itself is clearly invalid for the task. Use a soft control when repetition is a quality signal rather than a correctness rule.

For example, exact repeated phrases may be unacceptable in a short marketing description, while exact reuse can be essential in source code, JSON keys, citations, dialogue, or technical documentation.

Sampling settings can interact with repetition

Repetition does not happen in isolation from the rest of decoding. Temperature, top-k, top-p, greedy decoding, beam search, and other generation choices affect which alternatives remain competitive.

For example, if a model’s distribution is extremely concentrated on a looping continuation, mild sampling may keep returning to the same tokens. Increasing randomness can sometimes expose alternatives, but it can also reduce coherence. A repetition penalty changes scores more directly, while a hard constraint removes selected continuations entirely.

Do not assume that one control can compensate safely for another. A highly restrictive candidate set combined with aggressive repetition rules may leave the decoder choosing among poor alternatives. Exact behavior when constraints become difficult to satisfy depends on the generation implementation.

Tune related decoding settings together and measure the complete configuration.

Measure repetition at the level users experience

A good evaluation needs both a repetition metric and a quality check. Optimizing only for fewer duplicate n-grams can reward outputs that avoid repetition by becoming vague, short, or incoherent.

For exact local repetition, one simple diagnostic is the fraction of generated n-grams that are duplicates. If a sequence contains ten 4-gram occurrences and two are repeats of earlier 4-grams, the duplicate-occurrence rate is:

2 / 10 = 0.20

This is a diagnostic definition, not a universal standard. You must specify whether the denominator counts all occurrences, unique n-grams, or something else before comparing results.

Also track task-specific quality. Depending on the application, that can include:

  • correctness or human preference;
  • completion rate for required fields;
  • output length;
  • syntax or schema validity;
  • latency and generated-token cost;
  • the frequency of obvious phrase or sentence loops.

Manual inspection is particularly valuable. Two systems can have the same duplicate n-gram rate while one repeats an annoying sentence and the other merely reuses harmless technical phrases.

Diagnose the source before adding a penalty

Decoding controls are not the right fix for every repeated output.

The prompt requests repetition

A prompt may contain duplicated instructions, repeated examples, or a template that encourages the model to restate headings and boilerplate. Fixing the prompt is often cleaner than fighting it at decoding time.

Retrieved context contains duplicates

In retrieval-augmented generation, near-duplicate chunks can present the same passage several times. The model may then reflect that redundancy in its answer. Deduplicating or diversifying retrieval addresses the input problem rather than masking it after generation.

The stopping condition is wrong

A model may continue generating after the useful answer is complete because the application allows an unnecessarily large token budget or fails to recognize a task-specific stopping condition. Better termination can remove the opportunity for a late repetition loop and save inference work at the same time.

The task naturally repeats exact text

Code generation, structured data, translation, and document transformation often require faithful reuse. A global repetition rule can damage these tasks. If only one field or free-text section is problematic, apply controls at the narrowest layer your generation system supports.

Common mistakes

Treating all repetition as bad

A low repetition score is not the objective. Useful output is the objective. Evaluate whether repeated content is harmful before suppressing it.

Turning a soft preference into a hard ban

A hard n-gram constraint can make a loop impossible, but it can also prevent the only correct continuation. Prefer hard constraints for patterns that are genuinely invalid, not merely aesthetically undesirable.

Copying parameter values between systems

Generation parameter names are not universal specifications. Two serving systems can implement similarly named repetition controls with different score transformations, ranges, defaults, or interactions. Verify the documentation for the implementation you actually deploy.

Evaluating only short examples

Repetition failures often appear after a model has generated for some time. An evaluation set made only of short outputs can miss the behavior that motivated the control. Include realistic output lengths and difficult prompts.

Ignoring tokenization

N-gram constraints operate on tokens in many generation systems, not necessarily words. A phrase’s token boundaries depend on the tokenizer. Reason about the actual token sequence when an apparently simple constraint behaves unexpectedly.

When to use each approach

Start without a repetition control if your evaluation does not show a meaningful repetition problem. Every additional decoding rule adds behavior that can interact with the model and task.

When exact loops are rare but noticeable, first check prompts, retrieved context, and stopping behavior. If the model still overuses previously generated tokens, a modest soft penalty can be a reasonable experiment because it discourages reuse without categorically banning it.

Use a hard no-repeat n-gram rule when repeated token spans themselves violate the output requirement and testing shows that the constraint does not block legitimate text. It is a stronger intervention and deserves stronger validation.

For applications with multiple output types, avoid assuming one global setting is appropriate. Free-form prose, code, JSON, and quotations have different repetition requirements even when they use the same underlying model.

Conclusion

Repetition control works by changing decoding decisions based on generation history. Soft penalties make reused tokens less attractive; hard n-gram constraints make selected repeated patterns impossible. Neither approach can decide whether repetition is meaningful in the way a human reader can.

The practical strategy is to diagnose the source, apply the weakest control that addresses the observed failure, and evaluate both repetition and task quality. A generation that repeats less is only an improvement when it also remains correct, coherent, and faithful to the job it was asked to do.

Artificial Intelligence LLM Sequence Generation Inference