Autoregressive generation normally commits one token before the next target-model step can be evaluated. Speculative decoding changes that execution pattern. A cheaper proposal distribution produces a short candidate block, while the target model evaluates the proposed positions together. An acceptance rule then determines how much of that block can become output.
The mechanism is not ordinary batching. Tokens inside the proposal remain autoregressive, and the target model still defines the intended output distribution when an exact speculative-sampling algorithm is used. The useful change is that one expensive verification pass can account for several output positions when enough proposals are accepted.
Drafting moves serial work to a cheaper model
Let the target distribution at a position be (p(x)) and the proposal distribution be (q(x)). The proposal model samples candidate tokens sequentially because each proposed token becomes context for the next proposal. If it drafts (k) tokens, the target model can evaluate logits for those candidate positions in a single forward pass with a causal mask.
This does not make the target computation independent across positions. Transformer verification can process the whole candidate block because all proposed prefix tokens are already available as inputs to that pass. Causal attention still prevents a position from reading later positions.
The draft model therefore needs to be cheap enough that its serial proposal work costs less than the target steps it can eliminate. A draft that closely tracks the target distribution can increase acceptance, but agreement alone does not determine end-to-end latency. Draft cost, verification cost, block length, batch shape, memory traffic, and runtime kernels also matter.
Exact sampling needs a correction rule
Accepting a proposal merely because it has high target probability would generally alter the target distribution. Exact speculative sampling uses an acceptance-and-correction construction instead.
For a proposed token (x) sampled from (q), one standard acceptance probability is
[ a(x) = \min\left(1, \frac{p(x)}{q(x)}\right). ]
If the proposal is rejected, a replacement can be sampled from the normalized positive residual
[ r(x) \propto \max(0, p(x)-q(x)). ]
This construction compensates for regions where the proposal assigns more probability than the target. With the complete algorithm and its stated assumptions, the resulting token distribution matches target-model sampling rather than approximating it.
That property belongs to the acceptance algorithm, not to the phrase speculative decoding in general. Systems may use deterministic verification, relaxed acceptance, tree candidates, auxiliary heads, or other variants. Such methods need their own distributional claim; the exact-sampling result cannot be transferred automatically.
Rejection truncates the accepted prefix
Verification is ordered. Suppose a draft proposes (x_1, x_2, \ldots, x_k). If the first three tokens pass their acceptance tests and the fourth is rejected, later draft tokens were conditioned on a prefix that will no longer exist after correction. They cannot simply remain committed.
The decoder keeps the accepted prefix, emits the correction token required by the algorithm, and starts another speculative round from the resulting context. This prefix property makes acceptance length a central runtime quantity: a verification pass is most useful when it commits several positions.
A longer draft block offers more possible accepted tokens per target pass, but it also exposes more positions at which rejection can stop the block. There is no context-independent block length that follows from the mechanism alone. Practical selection depends on proposal quality and serving costs.
Verification changes the cache path
Speculative execution also affects KV-cache handling. The target pass computes states for candidate positions before the runtime knows which candidates will survive. After verification, cache entries corresponding to discarded suffix tokens must not become part of the committed decoding state.
An implementation can handle this with cache truncation, staged writes, index management, or another runtime-specific strategy. The semantic requirement is simpler than the storage design: the next decoding round must observe target-model state corresponding only to the committed prefix.
The draft model has its own state as well. If it uses a separate autoregressive model, its cache must be advanced or rebuilt consistently with the accepted prefix and any correction token. A serving engine that optimizes target verification but repeatedly reconstructs draft state can lose part of the intended latency gain.
Acceptance rate is not the same as speedup
A high fraction of accepted draft tokens is useful, but it is not a complete performance metric. Speculation adds proposal computation, verification over multiple positions, acceptance logic, and cache bookkeeping. The target pass may also have different kernel shapes from single-token decoding.
The relevant comparison is end-to-end work per committed output token under the actual serving configuration. Hardware utilization, batch size, prompt and generation lengths, draft size, target size, and memory behavior can change the result. A configuration can accept many proposals yet deliver a modest latency change if drafting or verification overhead is large.
Conversely, the mechanism is most attractive when target-model decode steps are expensive, proposal generation is substantially cheaper, and several proposals commonly survive each verification round.
Speculative decoding therefore has a precise implementation boundary. It reduces the number of serial target-model decode rounds only when proposed continuations can be verified and committed in blocks. Exact output-distribution preservation requires the corresponding exact acceptance-and-correction procedure; parallel verification by itself provides no such guarantee.