A language model can assign high probability to a token for several reasons. Some reflect context-specific structure; others reflect generic tendencies that also appear in a weaker model. Contrastive decoding separates those signals by scoring candidate tokens with two models rather than one.
The larger model acts as an expert. A smaller or otherwise weaker model acts as an amateur. Generation favors tokens that the expert supports more strongly relative to the amateur, subject to a plausibility constraint from the expert distribution.
This is an inference-time mechanism. It does not change model weights, and its behavior depends on the relation between the two models as much as on either model in isolation.
Relative scores change the ranking
Let p_e(x | h) denote the expert probability for token x given history h, and let p_a(x | h) denote the corresponding amateur probability. A basic contrastive score can be written as:
s(x) = log p_e(x | h) - alpha * log p_a(x | h)Here, alpha controls the strength of the amateur penalty. With alpha = 0, the score reduces to the expert log probability. Increasing alpha gives more influence to differences between the models.
The subtraction changes token ordering. A token that receives a high score from both models can lose rank because the amateur also predicts it confidently. A token supported strongly by the expert but less strongly by the amateur can move upward.
The mechanism therefore does not simply ask whether the amateur dislikes a token. It asks how expert support compares with amateur support on the same prefix.
Expert plausibility constrains the comparison
Pure subtraction can promote a token that both models consider improbable if the amateur assigns an even smaller probability. That outcome is mathematically consistent with the score but undesirable for generation.
Contrastive decoding addresses this with an expert plausibility set. One common form retains tokens whose expert probability is at least a fraction of the expert model’s maximum token probability:
V_valid = {x : p_e(x | h) >= beta * max_z p_e(z | h)}with beta between 0 and 1. The contrastive score is then applied only inside V_valid.
This separates two roles. The expert determines which tokens remain credible under the current context. The expert-amateur difference ranks candidates within that region. Raising beta narrows the candidate set toward tokens already favored by the expert; lowering it gives the contrastive score more room to alter the result.
The exact plausibility rule is part of the decoding algorithm, not a universal property of two-model decoding. Implementations need to state the rule explicitly because a different threshold definition can produce a different candidate set even with identical model logits.
Model compatibility matters
Both models must score a comparable next-token event for direct token-level subtraction to make sense. Shared tokenization is the simplest case: token index i denotes the same token in both output distributions.
Different vocabularies complicate the operation. A token from one model may correspond to several tokens in the other, so subtracting logits by index has no semantic basis. A system can introduce a mapping or compare scores over a shared representation, but that becomes a different implementation with additional assumptions.
Prefix handling also has to remain aligned. If chat templates, special tokens, or normalization rules differ, the models may be scoring different effective contexts even when the visible prompt is identical.
The amateur does not need to be a compressed copy of the expert, but its relation to the expert determines what the subtraction removes. An amateur with unrelated behavior can penalize features that have little connection to generic expert tendencies.
Logit scales can alter penalty strength
Model logits are not calibrated to a common absolute scale merely because the models share a vocabulary. Applying softmax temperature or another logit transformation changes the resulting log probabilities and therefore changes the contrastive difference.
Suppose the expert distribution is sharp while the amateur distribution is relatively flat. The amateur term then varies less across candidate tokens, so it has less effect on ranking at a fixed alpha. A sharper amateur distribution can create larger token-to-token penalties.
For this reason, alpha cannot be interpreted independently of the score distributions produced by the model pair. Replacing either model can change the effective penalty even when the decoding parameter remains unchanged.
A useful implementation check is to inspect the three quantities separately for selected prefixes: expert log probability, amateur log probability, and the resulting contrastive score. Looking only at the final token hides whether ranking changes came from expert uncertainty or an unusually strong amateur penalty.
Contrastive scoring adds inference work
Ordinary autoregressive decoding needs a forward pass from the generating model for each decoding position, apart from batching and cache reuse details. Contrastive decoding also requires amateur scores for the same prefix.
The smaller model may require much less computation than the expert, but its work is not free. Both models also maintain state across generation when KV caching is used. The extra memory and compute depend on model sizes, cache formats, sequence length, batching, and the serving implementation.
This differs from speculative decoding. Speculative decoding uses a draft model to propose tokens that a target model verifies, with a design goal of reducing target-model decoding work while preserving the target distribution under its acceptance procedure. Contrastive decoding uses the second model directly in the token scoring rule, so the resulting generation policy intentionally differs from decoding the expert alone.
The distinction matters when evaluating latency and output behavior. A second model can serve very different algorithmic roles even if both systems appear to pair a large model with a smaller one.
Evaluation needs the expert-only baseline
A contrastive configuration changes the decoding policy, so comparison against the same expert with the same surrounding generation settings is the useful baseline. Changing temperature, candidate filtering, prompt formatting, or stopping rules at the same time makes attribution difficult.
Token-level traces can expose failure modes that aggregate output inspection misses. For a small set of representative prefixes, recording the expert rank, amateur rank, plausibility decision, and final contrastive rank shows whether the amateur is suppressing broad high-probability patterns or unexpectedly overriding context-specific expert preferences.
The method is most interpretable when the amateur has a clear role and the plausibility constraint keeps selection anchored to the expert. Without those conditions, the subtraction is still computable, but its effect becomes harder to connect to a specific model behavior.