A language model can assign high probability to text that is fluent but unhelpfully generic, repetitive, or too close to an easy pattern. Changing temperature or top-p changes how tokens are sampled from one model’s distribution, but it does not ask a different question: which candidate tokens are especially characteristic of a stronger model rather than a weaker one?

Contrastive decoding asks exactly that. It uses two language models at inference time: a stronger expert and a weaker amateur. A candidate is favored when the expert scores it well relative to the amateur, while a plausibility constraint prevents the decoder from choosing bizarre tokens merely because the amateur dislikes them even more.

The method is useful as a mental model even if you never deploy it. It separates two signals that ordinary decoding combines: whether a continuation is plausible according to a capable model, and whether that continuation reflects behavior the capable model has learned beyond a weaker baseline. This article builds that idea from a small example, then explains the constraints, costs, and failure modes that matter in practice.

Start with what ordinary decoding sees

Suppose an expert language model has produced the following next-token probabilities after a prefix:

candidate    expert probability
---------    ------------------
"the"        0.40
"a"          0.25
"specific"   0.20
"blue"       0.10
"therefore"  0.05

Greedy decoding chooses "the" because it has the highest probability. Sampling may choose another token, depending on the sampling rule and random draw, but every decision still comes from the expert distribution alone.

Now imagine that a much smaller model also strongly prefers "the" and "a", while it assigns relatively little probability to "specific". That difference can carry information. The generic tokens may be easy for both models, whereas "specific" may be a continuation that the stronger model supports much more confidently than the weaker model.

Contrastive decoding makes that difference part of the decoding objective.

Think in relative scores, not two votes

The expert and amateur do not vote independently and then take a majority. Instead, the decoder compares their scores for the same candidate token.

A simplified token-level score is:

contrastive_score(token)
    = expert_log_score(token) - amateur_log_score(token)

Using log probabilities makes the subtraction equivalent to a log probability ratio:

log P_expert(token) - log P_amateur(token)
= log(P_expert(token) / P_amateur(token))

So a token receives a large contrastive score when it is much more likely under the expert than under the amateur.

Consider this simplified example:

candidate    expert log p    amateur log p    difference
---------    ------------    -------------    ----------
"the"           -0.9             -0.7            -0.2
"a"             -1.4             -1.2            -0.2
"specific"      -1.6             -3.2             1.6
"blue"          -2.3             -3.0             0.7

If the decoder used only the difference, "specific" would win even though "the" is individually more probable under the expert.

This illustrates the purpose of the contrast: it can suppress modes that both models find easy and promote candidates that distinguish the expert from the amateur.

It also exposes a serious problem. A very unlikely token can have a huge ratio if the amateur assigns it an even smaller probability. Contrastive decoding therefore needs more than subtraction.

Keep implausible tokens out of the competition

The original contrastive-decoding formulation includes a plausibility constraint. Before applying the contrastive objective, it restricts attention to tokens that the expert itself considers sufficiently plausible.

One common way to express the candidate set is:

V_candidate = {
    token:
    P_expert(token) >= alpha * max_token P_expert(token)
}

Here alpha is a threshold between 0 and 1. If the expert’s most probable token has probability 0.40 and alpha = 0.25, a candidate must have probability at least:

0.25 * 0.40 = 0.10

For the earlier example, these tokens remain eligible:

"the"       0.40
"a"         0.25
"specific"  0.20
"blue"      0.10

A token with expert probability 0.001 would be excluded even if the amateur assigned it probability 0.0000001 and therefore produced an enormous contrastive ratio.

The two parts of the method now have distinct jobs:

expert plausibility filter -> which tokens are reasonable enough to consider
contrastive score          -> which reasonable token best separates expert from amateur

That separation is central. The amateur is useful for penalizing behavior, but it should not be allowed to make an expert-rejected token attractive simply by rejecting it more strongly.

The exact objective can vary

It is important not to treat one simplified equation as the only possible implementation. The original method and later variants use related but not identical parameterizations. Some formulations contrast log probabilities with temperatures; others work directly with expert and amateur logits and include a weight controlling the amateur penalty.

A commonly used logit-space form is conceptually:

score(token) = (1 + beta) * expert_logit(token)
               - beta * amateur_logit(token)

where beta controls the strength of the contrast.

The useful invariant is not a particular library API or constant. It is the structure of the decision:

  1. obtain next-token scores from an expert and an amateur for the same prefix;
  2. restrict candidates using the expert’s plausibility signal;
  3. prefer plausible tokens that score better under the expert relative to the amateur;
  4. append the selected token and repeat autoregressively.

If you implement a published formulation, follow that formulation’s exact equations and normalization rules rather than mixing parameters from different variants. Logits, log probabilities, temperature-scaled distributions, and weighted differences are related, but substituting one for another without re-deriving the objective can change token rankings.

Both models must score compatible tokens

The cleanest form of contrastive decoding assumes that the expert and amateur can score the same candidate vocabulary. In practice, this often means using models from a compatible family with the same tokenizer and token-to-ID mapping.

Why does this matter? Suppose token ID 314 means "database" for the expert but represents a different byte sequence for the amateur. Subtracting the two logits at index 314 would compare different events. The resulting score would have no useful interpretation.

Even tokenizers that produce similar-looking text are not automatically interchangeable. Different segmentation can turn one expert token into several amateur tokens. Supporting such pairs requires an explicit method for comparing probabilities over compatible text units; simple element-wise logit subtraction is not enough.

For a straightforward implementation, treat vocabulary compatibility as a precondition and verify it rather than assuming model names imply it.

Walk through one decoding step

Consider a prefix:

The deployment failed because

Assume the expert and amateur produce these illustrative probabilities:

candidate       expert    amateur
---------       ------    -------
"the"            0.36      0.42
"a"              0.22      0.28
"configuration"  0.18      0.04
"network"        0.12      0.08
"purple"         0.01      0.0001

The token "purple" has an impressive expert-to-amateur probability ratio:

0.01 / 0.0001 = 100

But that does not make it a good continuation. The expert itself assigns only 1% probability to it.

If alpha = 0.2, the plausibility cutoff is:

0.2 * 0.36 = 0.072

So "purple" is excluded. Among the remaining tokens, "configuration" stands out because the expert gives it substantially more support relative to the amateur:

0.18 / 0.04 = 4.5

By comparison:

"the":     0.36 / 0.42 ~= 0.86
"a":       0.22 / 0.28 ~= 0.79
"network": 0.12 / 0.08 = 1.5

This teaching example uses probability ratios because they are easy to inspect. A real implementation should compute the exact published score in a numerically stable form, normally using logits or log probabilities rather than dividing small floating-point probabilities directly.

After selecting a token, both models receive the updated prefix and the process repeats for the next position.

Why the amateur should be weaker in a useful way

Calling the second model an amateur can make it sound as if any bad model will work. That is not the right mental model.

The method relies on a useful relationship between the two distributions. The amateur should share enough language behavior with the expert that their scores are comparable, while exposing undesirable or simpler modes more strongly. If the amateur is unrelated, badly trained, or specialized for a different domain, the difference between the models may reflect irrelevant mismatch instead of a useful quality signal.

For example, pairing a general expert with an amateur trained almost entirely on source code could penalize ordinary prose tokens for reasons that have little to do with generation quality. The contrast would still produce numbers, but those numbers would answer the wrong question.

The gap in capability also matters. If the models are nearly identical, their scores may cancel and provide little extra signal. If they are too different, the amateur penalty can become noisy. The useful pairing is therefore an empirical choice, not a guarantee that smaller automatically means better as a contrast model.

The names are similar, but contrastive decoding and contrastive search are different techniques.

Contrastive decoding uses two language models and contrasts their token scores. Contrastive search typically uses one model and penalizes candidate continuations whose hidden representations are too similar to previous representations, with the goal of reducing degeneration while retaining model confidence.

A quick distinction is:

contrastive decoding -> expert distribution versus amateur distribution
contrastive search   -> model confidence versus representation similarity

Confusing the two can lead to incorrect implementation plans and misleading performance expectations.

Account for the inference cost

Ordinary decoding requires a forward step from one model per generated position. Basic contrastive decoding needs next-token scores from two models. That adds compute, memory pressure, and implementation complexity.

The amateur is usually smaller, so its cost can be much lower than the expert’s, but it is not free. Both models may need their own KV caches during autoregressive generation. Those caches grow with sequence length and consume device memory. If the additional model or cache causes memory pressure, device transfers, or less efficient batching, latency can worsen by more than the amateur’s raw parameter count suggests.

The practical cost depends on the serving setup:

  • model sizes and architectures;
  • whether both models fit on the same accelerator;
  • batch size and sequence length;
  • KV-cache memory use;
  • kernel and scheduling efficiency;
  • whether expert and amateur work can overlap;
  • the baseline decoding strategy being replaced.

Measure end-to-end latency and throughput on the deployment hardware. A quality improvement observed offline does not imply that the serving trade-off is acceptable.

Contrastive decoding also differs from speculative decoding. Speculative decoding uses a draft model to propose tokens that a target model verifies, with algorithms designed to preserve the target sampling distribution. Contrastive decoding intentionally changes the decoding objective by penalizing tokens favored by the amateur. The second model therefore serves a different purpose in each method.

Tune quality and cost with evaluation, not intuition

Two controls are especially important in common formulations: the plausibility threshold and the strength of the contrast.

A permissive plausibility threshold allows more candidates into the contrastive competition. This gives the contrastive signal more room to affect output, but it can admit tokens that the expert considers marginal. A restrictive threshold keeps generation closer to the expert’s high-probability region but can reduce the method to something closer to ordinary high-confidence decoding.

Similarly, a stronger amateur penalty gives the contrast more influence. Too little influence may produce almost no behavioral change. Too much can over-penalize ordinary tokens that both models correctly prefer.

Do not tune these parameters on a few attractive examples. Build an evaluation set representing the actual task and measure properties that matter for it. Depending on the application, that can include task accuracy, factuality, repetition, human preference, output length, format compliance, latency, and throughput.

Keep a baseline such as greedy, temperature sampling, or nucleus sampling under the same model and prompt conditions. Otherwise it is easy to attribute improvements to contrastive decoding that actually came from a changed prompt, random seed policy, stopping rule, or generation length.

Watch for failure modes

Contrastive decoding changes which plausible tokens are preferred; it does not make the expert knowledgeable about facts it never learned. If both models support the same false continuation, contrasting them may not reveal the error. The method is therefore not a general hallucination detector or factuality guarantee.

It can also suppress valid generic language. Articles, punctuation, connective words, and common phrases are often probable under both models because they are genuinely appropriate. A poorly tuned penalty can prefer unusual wording merely because it differentiates the expert from the amateur.

Domain shift creates another risk. A pairing and threshold tuned on open-ended prose may behave differently for code, structured output, mathematical reasoning, or a specialized technical domain. The amateur’s mistakes in one domain may not be useful negative signals in another.

Finally, deterministic token selection can hide sensitivity to small score changes. When two candidates have similar contrastive scores, minor numerical or model-version differences can flip the winner and alter the rest of an autoregressive continuation. Reproducibility therefore requires pinning model versions, tokenizer versions, decoding parameters, and relevant numerical settings just as it does for other generation methods.

Know when a simpler decoder is enough

Contrastive decoding is most worth testing when generation quality is limited by undesirable modes that a weaker related model exhibits more strongly, and when you can afford a second model at inference time. It is especially attractive as a training-free experiment because it changes decoding rather than requiring another fine-tuning run.

A simpler decoder is often preferable when:

  • latency or memory is tightly constrained;
  • you only have black-box access to one model’s generated text rather than token-level scores;
  • the expert and available amateur use incompatible vocabularies;
  • ordinary greedy or sampling-based decoding already meets quality requirements;
  • the task requires a distribution-preserving acceleration technique rather than a changed generation objective.

The last point is important. Contrastive decoding is a quality-oriented decoding rule, not a free speed optimization. If the primary problem is decoding latency, techniques such as batching, quantization, or speculative decoding address different bottlenecks and may be more appropriate depending on the serving system.

Keep the mental model simple

Contrastive decoding works by asking two questions at each generation step. First, which tokens does the expert consider plausible? Second, among those tokens, which ones are supported more strongly by the expert than by the amateur?

That gives a practical pipeline:

same prefix
   |-- expert -> plausibility + expert scores --|
   |-- amateur -------------> amateur scores ---|-> contrast -> next token

The plausibility filter prevents a low-quality amateur from promoting nonsense through an extreme score difference. The contrast then acts inside that plausible region, suppressing continuations that look disproportionately attractive to the weaker model.

The method can improve generation in suitable model pairs, but its value is empirical. Treat the amateur as an additional inference-time signal, verify vocabulary compatibility, measure the extra serving cost, and compare quality against simpler decoding baselines on the task you actually need to solve.

Artificial Intelligence LLM Inference Text Generation