A language model can assign high probability to a token for two different reasons: the token may fit the prompt particularly well, or it may simply be common under many contexts. Contrastive decoding tries to separate those effects by comparing the next-token distributions of two models. A stronger expert supplies the main distribution, while a weaker amateur supplies a signal for patterns that do not require the expert’s extra capability.

The method changes token ranking at inference time. It does not retrain either model, and it is distinct from speculative decoding: both methods can use a large and a small model, but their scoring objectives and compute paths are different.

The score rewards an expert-amateur probability gap

Let p_e(v | x) be the expert probability for candidate token v after prefix x, and let p_a(v | x) be the amateur probability for the same token. A basic contrastive score is:

s(v | x) = log p_e(v | x) - log p_a(v | x)

The subtraction changes the ranking. A token that both models consider very likely receives less contrastive advantage than a token strongly favored by the expert but not by the amateur.

Consider three candidate tokens:

token    expert p    amateur p    log(expert) - log(amateur)
A        0.40        0.35         0.134
B        0.25        0.08         1.139
C        0.02        0.0001       5.298

Token B can outrank A even though its expert probability is lower. That is intentional: the score is not asking for the expert’s most probable token in isolation.

Token C exposes a separate problem. Its ratio is enormous because the amateur assigns almost no probability to it, yet the expert itself considers the token unlikely. Pure subtraction can therefore promote a token that is contrastive but implausible.

A plausibility constraint keeps the expert in control

Contrastive decoding addresses the low-probability problem by restricting scoring to tokens that remain plausible under the expert. One form uses a threshold relative to the expert’s highest next-token probability:

V_valid(x) = {
    v : p_e(v | x) >= alpha * max_w p_e(w | x)
}

for a chosen alpha between zero and one. Tokens outside V_valid are excluded from selection.

With alpha = 0.1 and a maximum expert probability of 0.40, the cutoff is 0.04. Token C from the earlier example is removed before its large contrastive score can dominate. Tokens A and B remain eligible.

This mask gives the expert distribution two roles. It determines which tokens are credible enough to consider, then participates in the contrastive score used inside that set. The amateur can alter ordering among plausible candidates, but it does not get unrestricted power to rescue tokens that the expert places far into its tail.

The threshold is a real behavior control rather than a cosmetic constant. A smaller value admits more of the expert tail, giving the contrast term more room to change generation. A larger value confines selection closer to high-probability expert tokens. The useful setting depends on the two distributions and the generation task; there is no model-independent value that preserves the same candidate set across prompts.

The amateur is a reference distribution, not a draft generator

The smaller model in contrastive decoding has a different job from the draft model in speculative decoding.

Speculative decoding uses a draft model to propose tokens and a target model to verify them with an acceptance procedure that preserves the target distribution when the algorithm’s stated conditions are met. Its main objective is reducing the cost of autoregressive generation.

Contrastive decoding evaluates expert and amateur distributions to define a new decoding score. The resulting token distribution is deliberately different from ordinary expert-only decoding. The amateur is useful because its probabilities form the reference being subtracted, not because its proposed sequence can be accepted cheaply.

That distinction affects implementation. Contrastive decoding needs aligned next-token scores from both models at each generation position used by the method. If the models have incompatible token vocabularies, direct token-by-token subtraction is not defined without an additional mapping or a different formulation. Sharing a tokenizer makes the basic operation straightforward, but shared tokenization alone does not guarantee that the contrast signal is useful.

Amateur strength changes what gets canceled

The contrast term has meaning only relative to the chosen amateur. If expert and amateur distributions are nearly identical, subtracting their log probabilities leaves small differences and provides little separation. If the amateur is extremely weak, its distribution may reflect errors unrelated to the behavior the developer wants to suppress.

The useful contrast comes from capability differences that appear in the next-token distributions. The amateur should still produce a coherent reference distribution over the same candidate tokens. Its role is not to be maximally bad.

This also means model size is a proxy, not the mathematical requirement. The scoring rule only sees two distributions. A smaller model is a convenient way to obtain a weaker reference, but other constructions can create a contrast signal as long as the semantics of that signal are understood.

A developer evaluating an amateur choice should therefore inspect more than final text. Token-level diagnostics can reveal whether the selected tokens gain score because the expert favors them or merely because the amateur assigns unusually tiny probability. The plausibility mask limits the second case, but it does not make every remaining contrast meaningful.

Logit subtraction requires careful normalization

Implementations often have logits rather than probabilities. For model logits z_e and z_a:

log p_e(v) = z_e(v) - logsumexp(z_e)
log p_a(v) = z_a(v) - logsumexp(z_a)

Subtracting the log probabilities gives:

log p_e(v) - log p_a(v)
= z_e(v) - z_a(v)
  - logsumexp(z_e) + logsumexp(z_a)

The final two terms are constant across candidate tokens at one decoding position. If the operation is only an argmax over the same candidate set, subtracting raw expert and amateur logits produces the same ranking as subtracting their normalized log probabilities.

That equivalence has boundaries. It concerns ranking at a fixed position with the same vocabulary and no token-dependent transformation added between the two forms. If code converts scores into probabilities, combines them with other terms, applies model-specific temperatures, or compares values across positions, the dropped normalization constants can matter to the surrounding computation.

Temperature also changes the contrast. Dividing logits by different temperatures scales each model’s contribution before subtraction:

s(v) = z_e(v) / T_e - z_a(v) / T_a

This is no longer just a neutral numerical rewrite of the equal-temperature score. It changes how strongly each model affects candidate ordering.

Extra inference cost is part of the mechanism

Ordinary autoregressive decoding runs the target model for each generated position. Contrastive decoding additionally needs the amateur distribution at those positions. A smaller amateur can cost much less per token than the expert, but its execution is still additional work and state.

Both models also maintain their own autoregressive state when KV caching is used. The memory footprint therefore includes cache state for each model, plus their parameters and temporary activations according to the serving implementation.

Some systems can overlap portions of the two forward passes or place the models on different devices. Those are implementation choices, not properties guaranteed by the decoding rule. The core method specifies how scores are contrasted; it does not guarantee a latency profile.

This cost profile is another reason not to treat contrastive decoding as a drop-in speed technique. It changes the generation objective and pays extra inference work to obtain the reference distribution.

Evaluation must match the changed objective

A contrastive decoder can produce text that differs from greedy, sampling, or beam decoding from the same expert because it is optimizing a different local score. Comparing only expert log probability can therefore be misleading: a contrastive choice may intentionally have lower expert probability than the expert-only choice while still passing the plausibility constraint.

Useful evaluation should keep the decoding configuration explicit: expert model, amateur model, tokenizer compatibility, plausibility threshold, temperatures, and any later sampling rule all affect the result. Changing the amateur is not equivalent to changing a minor sampler parameter; it changes the reference distribution used at every contrasted position.

The central boundary is simple. Contrastive decoding can emphasize tokens whose support is stronger in an expert than in a weaker reference, but that signal is only as meaningful as the relation between those models. The plausibility constraint prevents the probability ratio from becoming the sole authority. For developers, the main design problem is therefore not the subtraction itself, but choosing a reference model and candidate constraint that encode a useful distinction without letting tail probabilities dictate generation.

Artificial Intelligence Language Models Decoding Inference