Autoregressive decoding normally commits one token after each target-model evaluation. A sequence of K generated tokens therefore creates a serial dependency chain: token t+1 cannot be sampled until token t is fixed and becomes part of the prefix. Speculative decoding changes the amount of useful work obtained from a target-model call without removing that causal dependency.
A cheaper draft model first proposes several continuation tokens. The target model then evaluates those candidate positions together. Tokens whose draft probabilities are compatible with the target distribution can be accepted, while the first rejected position is corrected using a residual distribution. The resulting samples follow the target model’s distribution when the acceptance and correction procedure is implemented as specified.
This property separates speculative decoding from approximate decoding. The draft model influences execution cost and acceptance frequency, but it does not become the authority for the final probability distribution.
Drafting moves serial work to a cheaper model
Let p denote the target model distribution and q the draft model distribution. Starting from an existing prefix, the draft model generates a block such as
x1, x2, x3, x4and records the conditional draft probability for each sampled token. These proposals are still autoregressive inside the draft model: each proposed token extends the prefix used to produce the next proposal.
The useful asymmetry is computational. If q is substantially cheaper to evaluate than p, several serial draft steps can cost less than the corresponding number of target-model steps. The target model receives the original prefix plus the proposed continuation and computes scores for the candidate positions in one batched evaluation.
Transformer evaluation over a supplied token block can produce logits for multiple positions concurrently because all block tokens are already present as inputs. Causal masking still prevents a position from attending to future positions. Parallel verification therefore does not remove causal semantics; it changes when candidate tokens become available to the expensive model.
Acceptance depends on both target and draft probabilities
A proposed token cannot simply be accepted whenever it has high target probability. Exact speculative sampling uses a probability ratio. For a draft token x sampled from q, the acceptance probability is
min(1, p(x) / q(x))where both distributions are conditioned on the same accepted prefix at that position.
If the target assigns at least as much probability to x as the draft model does, the token is accepted with probability one. If the draft model overweights x relative to the target, acceptance is reduced proportionally.
This rule compensates for the fact that candidate tokens came from q, not p. Accepting every draft token that appears plausible would generally bias output toward the draft distribution.
Acceptance proceeds from left to right. Once a candidate is rejected, later draft tokens were generated from a prefix containing a token that is no longer accepted. Those later candidates cannot be committed as if their conditioning prefix were still valid.
Rejection uses a residual distribution
At the first rejected position, sampling directly from p would not by itself preserve the target distribution, because the rejection event already contains information about the draft sample. The correction distribution accounts for that conditioning.
For the basic speculative-sampling construction, the replacement token is drawn from a distribution proportional to
max(0, p(x) - q(x))across the vocabulary, followed by normalization. The accepted portion contributed through draft proposals and the residual portion contributed after rejection combine to recover the target distribution.
The exact algebra is part of the algorithm rather than an optional quality adjustment. Implementations that replace ratio acceptance or residual correction with heuristic thresholds can still be useful systems, but they no longer inherit the same distribution-preservation statement.
Numerical behavior also matters. Published formulations state the property for the specified probability operations; finite-precision kernels, altered sampling code, truncation, or inconsistent logits processing can create implementation-level differences. Distributional equivalence is an algorithmic property contingent on matching the required operations.
A fully accepted block yields more than one committed token
When every draft token in a block is accepted, the target evaluation contains enough information to sample an additional token from the target distribution after that block. A successful verification round can therefore commit the whole proposed block plus one target-sampled token.
If rejection occurs early, the round commits fewer draft tokens before correction. The acceleration potential is consequently tied to acceptance length, not merely to the configured draft-block size.
A larger block exposes more possible accepted tokens per target call, but it also spends additional draft computation and verification work on suffix positions that become unusable after an early rejection. The best block length is therefore workload- and implementation-dependent rather than an architectural constant.
Draft quality affects speed, not target authority
A draft model that closely tracks the target distribution tends to produce candidates with higher acceptance probability. That can increase the number of committed tokens obtained from each target evaluation. A poorly matched draft model can cause frequent early rejection and leave little opportunity to amortize target-model latency.
Closeness alone is insufficient for speed. The draft model must also be cheap enough that proposal generation does not consume the saved target time. Hardware utilization, batch size, sequence length, memory traffic, kernel launch overhead, and serving scheduler behavior all influence the realized latency.
This makes speculative decoding a systems optimization with a probabilistic correctness condition. Acceptance rate is useful, but it is not a standalone performance metric. Two configurations with the same acceptance pattern can have different end-to-end latency if their draft cost or verification efficiency differs.
Logit processing must be consistent across verification
Production decoders often transform raw logits before sampling. Temperature, vocabulary masks, repetition penalties, top-k selection, top-p truncation, or application-specific constraints can change the effective token distribution.
The p and q values used by speculative acceptance must correspond to the distributions from which target and draft decisions are actually made. Applying a transform during ordinary target sampling but omitting its effect from speculative verification changes the probability calculation. The same concern applies when a constraint makes some tokens impossible.
Not every decoding policy maps cleanly to the same speculative-sampling formulation. Deterministic greedy verification, constrained decoding, and stochastic sampling have different decision semantics even when implementations place them behind one generation API. A distribution-preservation claim should therefore name the decoding rule it covers instead of treating all generation modes as interchangeable.
Verification changes the unit of target-model progress
Ordinary autoregressive decoding usually obtains one committed token from one serial target step. Speculative decoding inserts a proposal-and-verification boundary: cheap serial work creates a candidate block, then one expensive evaluation can validate a variable-length prefix of that block.
The target model remains the final statistical authority. Rejection does not represent an error by the target model, and acceptance does not certify that a token is uniquely correct. Both are sampling events constructed so that execution can reuse draft proposals while retaining the target distribution.
The practical boundary is equally important. Speculation cannot guarantee a latency reduction merely from the presence of a smaller model. It pays when accepted progress per expensive verification outweighs draft generation, probability bookkeeping, rejected suffix work, and serving overhead. The mechanism preserves target sampling semantics; the speedup remains an empirical property of the complete inference stack.