Autoregressive generation normally asks the target model to produce one next-token distribution at a time. Speculative decoding changes that execution pattern. A cheaper draft model proposes several tokens, then the target model evaluates those candidates in a batch and decides how much of the proposal can be accepted.
The useful property is not merely that two models participate. The verification rule determines whether the optimization preserves the target model’s intended decoding distribution or silently changes it.
Draft tokens are proposals, not final output
Let the draft model assign a next-token distribution q and the target model assign p. A draft token sampled from q is only a candidate. Accepting every proposed token would generate from the draft model for those positions, so output behavior would generally differ from ordinary target-model sampling.
A distribution-preserving speculative sampler instead checks each proposed token using information from both distributions. One common acceptance probability for a proposed token x is:
min(1, p(x) / q(x))If a proposal is rejected, the replacement sampling rule must account for the probability mass already represented by the rejected proposal process. The exact algorithm matters. Replacing a rejected token by simply sampling from p is not interchangeable with the correction rule used by distribution-preserving speculative sampling.
This distinction separates speculative decoding as an inference optimization from approximate decoding methods that intentionally alter the output distribution.
Verification can amortize target-model work
The target model still has to score the proposed positions. The opportunity comes from evaluating several positions in one target-model invocation rather than advancing the target model through the same number of serial decoding calls.
Suppose the draft model proposes K tokens. The target model can process the current prefix plus those candidates and produce distributions needed to verify the proposal. If several consecutive draft tokens are accepted, generation advances by several positions after that target evaluation.
The gain is therefore tied to execution cost, not to eliminating target-model computation entirely. Draft generation adds work, target verification adds work, and rejected proposals waste part of the draft effort. Hardware utilization, model sizes, batch shape, memory traffic, and implementation details all affect the resulting latency.
A smaller draft model is useful only if its proposal cost is low enough relative to the target work it helps amortize.
Acceptance is sequential even when scoring is batched
Target scoring for a proposal block can be parallelized across candidate positions, but acceptance has a prefix dependency. Once a proposed token is rejected, later draft tokens were conditioned on a prefix that is no longer the generated prefix.
Those later tokens cannot simply remain accepted because their conditioning context has changed.
This gives speculative decoding an asymmetric shape: probability computation for several positions can happen together, while the logical validity of the candidate block is resolved from left to right. An implementation that ignores this boundary can reuse tokens whose model context no longer matches the emitted sequence.
The first rejection therefore limits the useful accepted prefix for that proposal block.
Draft quality changes useful proposal length
A draft model that closely tracks the target distribution tends to produce proposals that survive verification more often. A poorly matched draft model can be inexpensive per token yet offer little benefit because proposal blocks are frequently cut short.
Acceptance rate alone is not a complete performance metric. A larger draft model may increase acceptance while consuming more compute. A smaller one may propose cheaply but trigger more target verification per emitted token.
Proposal length creates a similar tension. Larger blocks offer more possible progress from one verification pass, but they also create more draft work that can become unusable after an early rejection. Short blocks cap wasted draft computation but expose less parallel target work.
The useful operating point depends on the target, draft model, prompts, decoding settings, and serving hardware. It is not a fixed property of speculative decoding as a mechanism.
Sampling settings belong inside the correctness boundary
Temperature, token masking, truncation, and other sampling transformations change the distributions used for generation. Verification must operate on distributions that correspond to the intended target decoding policy.
For example, applying a token filter to one model distribution but not the other changes the probability relationship used by the acceptance rule. Similar issues appear when logits are transformed at different stages or when numerical handling differs between draft and target paths.
Greedy decoding has a different verification structure from stochastic sampling because the desired output is the target model’s selected token rather than a sample from its full distribution. An implementation can accept draft tokens while they match the target decisions and fall back when they diverge, but that procedure should not be conflated with stochastic acceptance and correction.
The decoding policy is therefore part of the algorithm, not a wrapper around it.
Tokenizers and vocabularies constrain pairing
A simple speculative decoder compares draft and target probabilities for the same token candidates. That assumes compatible token identities and sequence boundaries.
If the two models tokenize text differently, a draft token sequence does not map trivially onto target-token positions. Specialized methods can bridge different tokenizations, but ordinary token-level verification cannot treat mismatched vocabularies as if token IDs had shared meaning.
Even with a shared tokenizer, special-token handling and prompt formatting must remain consistent. A proposal generated from a different effective prefix is not a valid candidate for the target context merely because its visible text appears similar.
Model pairing is thus an interface constraint as well as a quality choice.
Performance evaluation needs emitted-token accounting
Requests per second can hide the mechanism that determines speculative-decoding gains. More diagnostic measurements include accepted draft tokens per proposal, target verification calls per emitted token, draft compute per emitted token, and end-to-end latency under matched output settings.
Comparisons should keep the target model and decoding policy fixed when the goal is to isolate inference acceleration. If output constraints or sampling parameters change at the same time, latency differences no longer measure only the speculative path.
Workload composition also matters. Prompt processing can dominate short generations, leaving little decode time to optimize. Long generations expose more opportunities for repeated proposal and verification cycles.
Speculative decoding is most useful to reason about as a scheduling transformation around target-model generation. The draft model predicts work that might be reusable; verification decides which predicted work is valid. Any implementation that measures speed without checking that boundary risks attributing gains to an optimization that has also changed the model’s output process.