Greedy decoding can keep selecting locally probable tokens even when the resulting continuation becomes repetitive. Sampling can break that pattern, but it does so by introducing randomness. Contrastive search takes a different route: it remains deterministic for fixed inputs and settings while scoring likely next-token candidates against a representation-level repetition penalty.
The method combines two signals that describe different properties of a candidate. The language-model probability favors tokens that fit the current prefix. A degeneration penalty disfavors candidates whose new hidden representation is too similar to representations already present in the generated context.
Candidate selection starts from model probability
At each decoding position, contrastive search first limits attention to a set of likely next tokens, commonly the model’s top k candidates. This restriction matters because the second part of the score is not intended to rescue arbitrary low-probability vocabulary items.
For each candidate, the decoder evaluates the hidden state that would result from appending that token. It then compares the candidate state with prior hidden states in the prefix.
A conceptual score can be written as
score(candidate) =
(1 - alpha) * model_probability(candidate)
- alpha * degeneration_penalty(candidate)The exact notation can vary across implementations, but the two terms express the core decision. alpha controls how much representation similarity can offset next-token probability.
When alpha is zero, candidate ranking reduces to the model-probability term within the candidate set. As alpha grows, hidden-state similarity has more influence on which candidate survives.
The degeneration penalty operates in representation space
The original contrastive-search formulation measures degeneration using cosine similarity between a candidate’s contextual representation and representations of earlier tokens. A common penalty takes the maximum similarity against the existing prefix.
Conceptually:
penalty(candidate) =
max cosine(candidate_hidden, previous_hidden_i)A candidate that produces a state closely aligned with an earlier state receives a larger penalty. This differs from a token-level repetition rule that blocks a token merely because its ID appeared before.
That distinction matters for subword models and paraphrastic text. Exact token repetition is a surface property. Hidden-state similarity is contextual and can react to repeated representational patterns even when the next token ID itself is not a duplicate.
The penalty is still only a proxy for undesirable degeneration. Similar representations can occur in valid text, and low similarity does not guarantee a coherent continuation.
Top-k and alpha control different boundaries
The candidate count k and weighting factor alpha affect separate parts of the search.
k defines which tokens are eligible for comparison. A small candidate set keeps the search close to the model’s highest-probability choices. Increasing it exposes more alternatives to the degeneration term, but also requires evaluating more candidate continuations.
alpha changes ranking inside that eligible set. A larger value gives the similarity penalty more leverage relative to model probability.
These controls interact. Raising alpha cannot select a token excluded from the top-k set. Raising k does not force a diverse token to win if the probability term and chosen alpha still favor another candidate.
Treating the two parameters as interchangeable can therefore produce confusing evaluations. One changes the search set; the other changes the score applied within it.
Hidden-state geometry affects the penalty
Cosine similarity is only informative to the extent that the model’s hidden representations provide useful angular separation for this purpose. If many token representations occupy a narrow region of the space, unrelated candidates can have high cosine similarity and the penalty loses discrimination.
The contrastive-search work connects this issue to representation anisotropy and pairs the decoding method with a contrastive training objective called SimCTG. Later use of contrastive search has also examined pretrained models whose representation spaces are sufficiently discriminative without that additional training.
For developers, the practical boundary is that the decoder is not operating only on output probabilities. Its behavior depends on internal hidden states. Two models with similar next-token distributions can react differently to the same search settings if their representation geometry differs.
This also means a runtime must expose the relevant hidden states or implement the scoring inside the generation stack. An API that returns only sampled text cannot reproduce the method externally.
Determinism does not remove search cost
Contrastive search avoids random sampling, but it requires more work than choosing the single largest next-token probability.
The decoder must consider multiple candidates and obtain a contextual representation for each candidate before ranking them. Implementations can batch or otherwise optimize those candidate computations, so the exact latency cost depends on the generation stack and model architecture.
The cost also scales with choices such as k. A larger candidate set increases the amount of candidate evaluation even if the final output still contains one token per decoding position.
This is a different systems profile from penalties that modify logits using only token history. A token-frequency penalty can often be computed from existing decoding state, while contrastive search needs model representations associated with candidate continuations.
Repetition metrics do not establish coherence
A decoder can reduce repeated n-grams and still produce text that drifts from the prompt. It can also preserve local fluency while making a poor factual or task-level choice. Contrastive search changes token selection; it does not add external evidence or correct knowledge stored in the model.
Evaluation should separate the failure mode targeted by the decoder from broader output quality. Repetition rate, continuation coherence, task constraints, and factual checks describe different properties.
Comparisons also need fixed prompts, model weights, stopping rules, and output-length handling. Otherwise a decoder that simply produces shorter text can appear to repeat less without demonstrating a better continuation policy.
Token blocking and contrastive search solve different problems
No-repeat n-gram constraints and token penalties act directly on generated symbols. They are straightforward when the requirement itself is lexical, such as preventing an exact phrase from appearing twice.
Contrastive search instead uses representation similarity as a signal while retaining likely candidates from the language model. It can discourage a broader form of local degeneration, but it does not provide a hard guarantee that a token or phrase will never repeat.
That boundary determines which mechanism fits an application. Hard lexical constraints are appropriate when repeated text is invalid by definition. Representation-aware ranking is more suitable when repetition is undesirable as a generation pattern but legitimate reuse must remain possible.
Contrastive search is therefore best treated as a decoding policy rather than a generic repetition filter. Its behavior comes from the interaction among model probability, candidate-set size, penalty weight, and hidden-state geometry. Changing any one of those pieces can alter the output even though the underlying model weights remain fixed.