Improve LLM Generation with Contrastive Decoding
A language model can assign high probability to text that is fluent but bland, repetitive, or overly driven by common patterns. Sampling adds variety, but increasing randomness can also admit weak continuations. Contrastive decoding takes a different route: compare a stronger model with a weaker reference model at each generation step, then favor tokens that the stronger model supports more distinctly.
The method changes decoding rather than model parameters. It can therefore be useful when you control inference for compatible models and want to experiment with generation quality without another training run. The extra model pass is not free, and the method needs a plausibility guard to avoid promoting strange tokens.
This article develops the core scoring idea, works through a small numerical example, and shows the engineering checks that matter before using contrastive decoding in a real system.
Start with two opinions about the next token
Assume an expert model and a smaller amateur model receive the same token prefix:
The database connection became unstable afterEach model produces a probability distribution over the next token. Imagine a tiny candidate set:
token expert amateur
the 0.40 0.38
deployment 0.30 0.08
a 0.20 0.18
midnight 0.10 0.01Choosing the expert’s highest-probability token gives the. That may be perfectly reasonable. Contrastive decoding asks a different question: which candidate is especially well supported by the expert relative to the amateur?
A basic contrastive score is:
score(token) = log P_expert(token | prefix)
- log P_amateur(token | prefix)For deployment, the expert-to-amateur probability ratio is 0.30 / 0.08 = 3.75. For the, it is only about 1.05. Since subtracting log probabilities is equivalent to taking the log of that ratio, deployment receives a stronger contrastive signal.
The intuition is simple. If both models strongly favor a generic continuation, agreement alone provides little evidence that the continuation reflects capabilities unique to the stronger model. A token favored much more by the expert can carry more useful signal.
That intuition has an important limit: a huge ratio can come from two tiny probabilities. A token with expert probability 0.0001 and amateur probability 0.000001 has a ratio of 100, yet the expert itself considers the token very unlikely. This is the reason contrastive decoding needs a plausibility constraint.
Keep the expert in charge of plausibility
The contrastive score should not search the entire vocabulary without restriction. Doing so can elevate tokens that the amateur dislikes even more than the expert does.
A practical procedure has two stages for each generated token:
- Use the expert distribution to form a set of plausible candidates.
- Apply the contrastive score only inside that set.
The original contrastive decoding formulation uses an expert-based plausibility constraint. Implementations can express such a guard as a probability threshold relative to the expert’s strongest candidate or through another candidate-selection rule that preserves the same principle: the amateur should help rank credible expert choices, not rescue candidates the expert considers implausible.
Consider this extension of the earlier example:
token expert amateur
deployment 0.300000 0.080000
midnight 0.100000 0.010000
zxq 0.000010 0.00000001The raw probability ratio for zxq is enormous. That does not make it a sensible continuation. An expert plausibility filter removes it before contrastive ranking.
This separation is central to the method:
expert distribution -> plausible set -> contrastive rankingThe expert decides what can reasonably be emitted. The expert-amateur difference decides which plausible candidate to prefer.
Contrastive decoding works in log space
Language-model inference normally exposes logits, which are unnormalized scores before softmax. For a model with logits z, the log probability of token i is:
log P(i) = z_i - logsumexp(z)For two models, the contrastive score becomes:
(z_expert_i - logsumexp(z_expert))
-
(z_amateur_i - logsumexp(z_amateur))Within one decoding step, both logsumexp terms are constants with respect to candidate token i. If you only need to rank candidates, they cancel as a shared offset:
rank_score_i = z_expert_i - z_amateur_iThis shortcut is valid only when token i refers to the same vocabulary item in both distributions and the objective uses the plain log-probability difference. If the models use different tokenizers or vocabulary indexing, subtracting logits by array position is meaningless.
The plausibility filter still needs information from the expert distribution or an equivalent expert-based rule. Logit subtraction does not replace that guard.
Tokenizer compatibility is a hard boundary
Two models can both be autoregressive transformers and still be unsuitable as a direct contrastive pair.
At every generation step, the expert and amateur must score the same candidate tokens for the same prefix representation. The cleanest setup uses models from a compatible family with the same tokenizer and vocabulary.
Suppose token ID 1842 means deployment for the expert but represents a different byte sequence for the amateur. Then:
expert_logits[1842] - amateur_logits[1842]compares unrelated events. The arithmetic runs, but the score has no useful interpretation.
Even matching vocabulary sizes do not prove compatibility. Check tokenizer identity, token-to-ID mapping, special tokens, and any preprocessing that changes the prefix. If those do not align, use a different decoding design rather than forcing direct token-level subtraction.
A minimal decoding loop
The following pseudo-code shows the essential control flow. It is deliberately model-API neutral:
prefix = prompt_tokens
while not stop(prefix):
expert_logits = expert(prefix)
amateur_logits = amateur(prefix)
candidates = plausible_tokens(expert_logits)
scores = expert_logits[candidates] - amateur_logits[candidates]
next_token = select(scores, candidates)
prefix.append(next_token)plausible_tokens is not decorative. Removing it changes the behavior materially.
select can choose the highest contrastive score for deterministic generation. A system can also build a probability distribution from adjusted scores and sample, but that adds another control surface such as temperature. If you do this, evaluate it as a distinct decoding configuration rather than assuming results from deterministic contrastive decoding transfer unchanged.
Both models must process the same growing prefix. With KV caching, each model can reuse its own cached attention state instead of recomputing the entire prefix at every step. You still maintain two model states, so memory and compute costs are higher than decoding with the expert alone.
Treat the amateur as a reference, not an oracle
The amateur model is useful because its distribution provides a baseline. It does not need to produce good text by itself, but it must provide a meaningful contrast.
If the amateur is too similar to the expert, their logits can be close across many tokens. The subtraction then provides little separation. If the amateur is too weak or behaviorally mismatched, its errors may be so broad that the contrast rewards accidental differences rather than useful expert capability.
Model size is therefore only one selection factor. Compatibility and behavior matter too.
A useful evaluation compares at least:
expert alone
expert + chosen baseline decoder
expert + contrastive decoderIf possible, also compare more than one amateur. This reveals whether an apparent gain comes from the contrastive principle or from a fortunate pairing.
Do not interpret the amateur as a detector of incorrect tokens. A low amateur score does not mean a token is factual, coherent, or safe. It means only that the amateur assigns less support to that token under the current prefix.
Cost changes at every generated token
Contrastive decoding adds work to the latency-critical generation loop. For every next token, you need logits from both models.
If the amateur is much smaller than the expert, its compute cost may be modest relative to the expert, but it is not zero. The actual latency impact depends on hardware, batching, memory bandwidth, model placement, cache handling, and whether the two forward passes can overlap.
There is also a memory cost. Each model needs parameters available for inference and typically maintains its own KV cache for the active sequence. Long contexts and large batches can make cache memory significant even when the amateur’s parameters are small.
Measure end-to-end behavior rather than estimating cost only from parameter counts. Useful production metrics include:
- time to first token;
- inter-token latency;
- tokens per second at representative concurrency;
- peak device memory;
- output quality on the target task.
A small quality gain may not justify a large throughput loss for an interactive service. For offline generation, the same trade-off can be acceptable.
Evaluate the behavior you actually need
Contrastive decoding was introduced for open-ended text generation. That does not make it a universal replacement for ordinary decoding.
Start evaluation with prompts that match the intended workload. If your system writes product descriptions, test product descriptions. If it produces technical explanations, test those. A generic text benchmark can miss failures tied to your prompt format or domain vocabulary.
Separate quality dimensions rather than collapsing everything into one impression. Depending on the application, inspect:
- repetition and degeneration;
- coherence across several paragraphs;
- relevance to the prompt;
- factual accuracy when source truth is available;
- formatting and instruction adherence;
- diversity across repeated generations;
- latency and resource use.
Keep generation length, stop rules, prompt templates, and other decoding settings controlled during comparisons. Otherwise a change in output can be incorrectly attributed to contrastive scoring.
Human review can be especially useful for open-ended text because two fluent outputs can differ in specificity, flow, and repetition in ways that a single automatic metric misses.
Common mistakes
Subtracting logits from incompatible vocabularies
This is the most fundamental implementation error. Token IDs must represent the same candidates in both models. Check the mapping explicitly instead of inferring compatibility from model names.
Omitting the plausibility constraint
Raw expert-minus-amateur scoring can reward a token simply because the amateur assigns it an even smaller value. Restrict ranking to credible expert candidates.
Calling the contrastive score a confidence score
The score is a relative preference between two model distributions. It is not a calibrated probability that a token is correct. Do not use its magnitude as a confidence threshold without separate validation.
Changing several decoding controls at once
Switching to contrastive decoding while also changing temperature, prompt wording, maximum length, and stop criteria makes the result hard to interpret. Change one major factor at a time during evaluation.
Ignoring service-level cost
A method that improves sample quality in a notebook can still be a poor deployment choice if it doubles model state, reduces batch capacity, or pushes inter-token latency beyond the product target.
Cases where a simpler decoder is enough
Contrastive decoding is most compelling when you can run two compatible models, generation quality is important enough to justify extra inference work, and baseline decoding shows a concrete problem worth addressing.
A simpler decoder is often preferable when the expert already meets the quality target, latency is strict, device memory is tight, or no suitable amateur model is available. Greedy decoding, temperature sampling, top-k selection, or nucleus sampling each has a smaller operational footprint because only one model distribution is required.
The method is also not a direct fix for missing knowledge. If the expert lacks required information, subtracting an amateur distribution does not add that information. Retrieval, better context, a stronger model, or task-specific adaptation may address that problem more directly.
Likewise, contrastive decoding is not a safety boundary. Any safety requirement still needs its own controls and evaluation.
Use contrast as a decoding hypothesis
Contrastive decoding offers a useful mental model for generation: common probability mass shared by a strong and weak model may be less informative than probability mass the stronger model supports distinctly. The expert-amateur difference turns that idea into a next-token ranking rule, while the plausibility constraint keeps the expert responsible for credible candidates.
For developers, the practical test is straightforward. Pair tokenizer-compatible models, keep an expert-based plausibility guard, benchmark against the expert’s normal decoder, and measure quality together with latency and memory. Keep the method only when that complete trade-off is favorable for the workload.