A language model can give high next-token probability to text that is fluent but generic. Contrastive decoding changes the ranking by asking for a second signal: does a weaker model also find the same candidate easy to predict? A candidate favored by the expert but not by the amateur receives stronger relative support than one both models score highly.
This is an inference-time mechanism. It does not alter either model’s parameters, and it does not convert the amateur model into a verifier. The decoder combines two token distributions and then selects from the resulting scores.
The score is a relative preference
Let the expert assign conditional probability p_E(x | c) to candidate token x given context c, and let the amateur assign p_A(x | c). A basic contrastive score can be expressed in log space as:
score(x) = log p_E(x | c) - alpha * log p_A(x | c)The coefficient alpha controls how strongly the amateur distribution affects the ranking. With alpha = 0, the score reduces to the expert log probability. Increasing alpha gives more influence to disagreement between the two models.
The subtraction has a specific interpretation. A token that is probable under both models can lose rank because the amateur explains much of its expert score. A token that receives strong expert support but weaker amateur support can rise. The method therefore ranks model disagreement, not expert probability alone.
That distinction also creates a failure mode. A token can obtain an attractive ratio simply because the amateur assigns it extremely low probability. Without another constraint, low-quality candidates from the expert’s tail can receive excessive contrastive scores.
Plausibility constraints protect the expert distribution
Contrastive decoding is commonly paired with a plausibility filter derived from the expert model. Instead of contrasting every vocabulary item, the decoder first keeps candidates whose expert probability is sufficiently close to the expert’s strongest candidate.
One relative form is:
V_valid = {x : p_E(x | c) >= beta * max_y p_E(y | c)}with beta between zero and one. Contrastive scoring is then applied only inside V_valid.
The filter and the contrastive score have separate jobs. The filter states that a token must remain plausible under the expert. The contrastive term then changes the order among those retained candidates. Tightening beta makes the decoder stay closer to the expert’s local preference; relaxing it gives the contrastive term a wider candidate set to rearrange.
This separation matters in implementations. Applying a large contrast coefficient without a plausibility boundary is not equivalent to using contrastive decoding with a broad but explicit expert filter. The former permits candidates that the expert itself may regard as poor continuations.
The amateur model defines the behavior being penalized
The amateur is not merely a cheaper copy used to save compute. Its distribution supplies the negative side of the comparison, so its capabilities determine what the contrastive score suppresses.
If the amateur closely matches the expert on a context, subtraction can produce little useful separation. If it is too weak, its probabilities may reflect errors that have little connection to the behavior the application intends to reduce. A useful contrast therefore depends on structured disagreement: the expert should represent distinctions that the amateur fails to capture.
Model pairing also creates a token-alignment requirement. The simplest token-level subtraction assumes both models score the same candidate identifiers over a compatible vocabulary. When tokenizers or vocabularies differ, logits cannot be subtracted position by position without an additional mapping or a different scoring construction.
Even with a shared tokenizer, raw logits from separate models are not directly comparable as calibrated absolute quantities. Formulating the contrast with log probabilities makes each model’s scores relative to its own normalization over the vocabulary before the two signals are combined.
Contrastive decoding adds inference work
Standard generation needs one model evaluation per decoding position, aside from cache and batching details. A two-model contrastive decoder needs scores from both expert and amateur for the active context.
The amateur can be smaller, so its forward pass may cost less than the expert pass, but it is still additional computation and state. Autoregressive serving also needs to manage a cache for each model when cached attention is used. The resulting memory and latency cost depends on model sizes, cache representation, hardware placement, batching, and the inference engine.
This cost profile differs from speculative decoding. Speculative decoding uses a draft model to propose tokens that the target can verify in groups, with algorithms designed to preserve the target distribution. Contrastive decoding instead changes token ranking by combining expert and amateur distributions. The second model serves a different mathematical role even when both techniques happen to use a smaller auxiliary model.
Temperature and truncation change the comparison
Contrastive scoring sits inside a larger decoding pipeline. Temperature scaling, top-k filtering, nucleus filtering, repetition adjustments, and hard token masks can all affect which scores reach token selection.
The order of these operations is not interchangeable in general. For example, filtering expert candidates before contrast prevents excluded tokens from returning because of a favorable amateur score. Filtering only after the two distributions are combined defines a different candidate policy.
Temperature also deserves explicit treatment. Scaling expert and amateur logits separately before softmax changes the sharpness of each distribution. Applying one scale to a final combined score is a different operation. An implementation should therefore define which representation is contrasted, when normalization occurs, and where any sampling temperature is applied.
These details are part of the decoding algorithm rather than incidental configuration. Two systems can both advertise contrastive decoding and still implement materially different ranking rules.
Token-level contrast does not verify factual claims
A high contrastive score indicates that the expert favors a candidate more strongly relative to the amateur under the chosen scoring rule. It does not establish that the token is factually correct, grounded in a source, or consistent with an external database.
The same boundary applies to longer generated statements. Re-ranking local token choices can change output quality, but the contrastive signal contains no independent evidence about the external world unless such evidence enters through some other mechanism.
Applications that require grounded claims still need a source of grounding and an evaluation method suited to that requirement. Contrastive decoding can shape generation behavior, but it should not be treated as a factuality certificate.
Parameter choices are coupled to the model pair
The contrast coefficient and plausibility threshold cannot be interpreted independently of the selected expert and amateur. Changing either model changes both probability distributions, which changes the scale and structure of disagreement seen by the decoder.
For that reason, a numeric setting copied from another model pair has no general guarantee of preserving the same behavior. Evaluation should examine the actual outputs and task constraints for the deployed pair, including cases where the expert and amateur strongly agree, strongly disagree, or assign low probability to the same candidate region.
The most useful mental model is not that the amateur tells the expert which token is bad. The decoder is constructing a new preference from two distributions. Its behavior is controlled by the quality of their disagreement, the expert plausibility boundary, and the exact score transformations applied before selection.