A draft model can propose a token that the target model would not have sampled from the same random draw, yet speculative sampling can still preserve the target model’s distribution. The key is not that the draft model predicts the target perfectly. Distributional correctness comes from the acceptance rule and the correction applied after rejection.
This separates two properties that are often grouped together. Draft quality controls how frequently proposals survive verification. The rejection-correction construction controls whether the resulting sample follows the target distribution.
Acceptance depends on a probability ratio
Let the target model define a next-token distribution (p(x)), while a faster draft model defines (q(x)) over the same token space. A proposal (x) is sampled from (q).
The canonical speculative sampling rule accepts that proposal with probability
[ A(x) = \min\left(1, \frac{p(x)}{q(x)}\right). ]
For a token with (q(x) > 0), the probability mass contributed by accepted draft proposals is therefore
[ q(x)A(x) = \min(q(x), p(x)). ]
That expression exposes the role of verification. If the draft assigns no more probability to a token than the target does, every proposal of that token can be accepted. If the draft overweights a token, only the portion supported by the target survives acceptance.
The accepted mass alone is not generally equal to (p). Rejected proposals leave target probability mass that still has to be supplied.
Rejection exposes a residual distribution
Define the positive residual
[ r(x) = \max(0, p(x)-q(x)). ]
After a rejection, the replacement token is sampled from the normalized residual distribution
[ p_{\mathrm{res}}(x) = \frac{\max(0, p(x)-q(x))} {\sum_y \max(0, p(y)-q(y))}. ]
This residual contains target mass that was not already covered by accepted draft proposals. Tokens for which the draft exceeds the target have zero residual mass; their supported target mass has already been represented by the accepted portion.
The total rejection probability is
[ 1 - \sum_x \min(p(x),q(x)). ]
For normalized discrete distributions, this is also equal to
[ \sum_x \max(0,p(x)-q(x)). ]
so the residual denominator is exactly the probability that correction is required.
Combining accepted mass with rejection followed by residual sampling recovers (p(x)). The draft distribution affects the path used to obtain the sample, but the final marginal distribution remains the target distribution when this rule is applied exactly.
Draft quality changes acceptance, not the target law
The overlap term
[ \sum_x \min(p(x),q(x)) ]
is the expected acceptance probability for a single proposal. A draft distribution close to the target has larger overlap and therefore causes fewer corrections. A poor draft can still participate in an exact sampler, but more proposals are rejected and the computational benefit can disappear.
This is an important systems boundary. A stronger draft is primarily a performance optimization under the exact scheme. It is not the source of the distributional guarantee.
The cost balance also includes draft-model latency, target verification cost, candidate length, memory traffic, batching behavior, and implementation overhead. Acceptance rate by itself does not establish end-to-end speedup.
Multiple draft tokens introduce an early-rejection boundary
Speculative decoding usually proposes a sequence rather than one isolated token. The draft model generates several candidate tokens autoregressively, and the target model evaluates probabilities for those positions in parallel where the architecture and runtime permit it.
Verification still proceeds in sequence because each accepted token determines the prefix for the next position. Once a draft token is rejected, later draft tokens were conditioned on a prefix that is no longer the active generated prefix. Those later candidates cannot simply be retained as if verification had succeeded.
The decoder instead applies the correction at the first rejected position and discards the unverified suffix for that speculative round. A later round begins from the corrected prefix.
If all proposed tokens in a block are accepted, canonical algorithms can sample an additional token from the target evaluation already available for the next position. This extra token is part of the throughput benefit: one expensive target verification can advance generation by more than the number of merely checked positions.
Greedy verification is a different contract
When generation uses greedy decoding, a runtime can compare draft tokens against target argmax tokens and retain a matching prefix. That mechanism can reproduce greedy target output under suitable deterministic execution, but it is not the same probabilistic construction as rejection-corrected sampling.
For stochastic generation, accepting a draft token merely because it falls among high-probability target candidates does not by itself preserve the target distribution. Likewise, replacing rejected tokens from the raw target distribution rather than the residual generally changes probability mass because some target mass has already been supplied through accepted proposals.
The exactness claim therefore belongs to a specific acceptance-and-correction algorithm, not to every system carrying a speculative-decoding label.
Numerical and runtime details still form an implementation boundary
The equations describe probability distributions in ideal arithmetic. Production inference operates with finite-precision logits, softmax implementations, random-number generators, token filters, and model-specific processors. If draft and target probabilities are transformed differently before verification, the distributions used by the acceptance rule no longer represent the same sampling contract.
Temperature, vocabulary masks, top-k filtering, nucleus filtering, repetition penalties, and other processors must be placed consistently with the algorithm implemented by the runtime. A system may deliberately use an approximate verifier for additional throughput, but then exact preservation of the original target distribution requires a separate proof for that modified rule.
The stable mechanism is narrower: draft proposals can reduce serial target-model work while exact rejection correction reallocates probability mass so accepted proposals and corrected rejections sum to the target distribution. Performance comes from cheap proposals and high acceptance. Distributional fidelity comes from the verifier mathematics.