Autoregressive generation normally invokes the target model once for every emitted token. Speculative decoding changes that execution pattern: a cheaper draft model proposes several tokens, and the target model evaluates the proposed block in a single verification pass. The speed opportunity comes from doing useful target-model work for multiple positions at once, not from treating draft output as authoritative.
Draft tokens are proposals, not final output
Let the target model define distribution p and the draft model define distribution q at a given position. The draft samples a candidate token from q. Verification then decides whether that candidate can be retained as a sample consistent with p.
For one proposed token x, the standard acceptance probability is:
a(x) = min(1, p(x) / q(x))If the proposal is accepted, decoding advances to the next drafted position. If it is rejected, the verifier samples a correction from the positive residual between the target and draft distributions:
r(x) ∝ max(0, p(x) - q(x))This correction is part of the distribution-preserving construction. Simply rejecting a draft token and sampling directly from p is a different procedure.
The exact algorithm also needs defined handling for zero draft probability and numerical normalization. Production implementations can organize these calculations differently, but distributional equivalence depends on preserving the acceptance and correction semantics of the chosen speculative-sampling method.
Verification can cover several positions in one target pass
A draft model can produce a sequence such as:
d1, d2, d3, d4The target model receives the prefix plus that draft block and computes target distributions for the relevant positions. Causal attention permits those positions to be evaluated together because each position only consumes preceding state.
Acceptance remains sequential. If d1 and d2 pass but d3 fails, d4 is not emitted from that speculative block. The accepted prefix is retained, a correction is sampled at the rejection point, and generation continues from the resulting sequence.
When every proposed token is accepted, the target computation can also provide a distribution for the token immediately after the draft block. Depending on the algorithm variant, that distribution can supply an additional emitted token before another draft phase begins.
Agreement controls useful speculative depth
A longer draft block does not automatically produce more useful work. Tokens after the first rejected position are discarded for that round, even if their individual predictions might otherwise have matched the target.
The useful block length therefore depends on agreement between q and p. A draft model that is cheap but poorly aligned with the target can create frequent early rejection. A more capable draft can raise acceptance but also consume more compute to produce each proposal.
This creates a serving-level optimization problem. Draft cost, target verification cost, batch shape, memory traffic, kernel behavior, and acceptance pattern all contribute to latency. A fixed number of draft tokens cannot establish a universal optimum across models and hardware.
Distribution preservation is conditional on the sampling procedure
Speculative decoding is sometimes described as using a smaller model to predict what a larger model would produce. That description misses a key boundary. The draft model does not need to reproduce the target model exactly; its proposals are filtered by a verifier constructed around the target probabilities.
For stochastic decoding, equivalence to ordinary target sampling requires the specified acceptance and residual-correction procedure. Replacing those operations with heuristic acceptance thresholds changes the resulting distribution unless a separate proof applies.
Greedy or otherwise deterministic variants have different conditions. Matching the target model’s selected token can be sufficient for a deterministic step, but that is not the same claim as preserving a stochastic target distribution.
Sampling controls also need to be consistent with the distributions used by the verifier. If temperature, truncation, penalties, or masking alter target probabilities, the verification calculation must operate on the corresponding effective distributions rather than an unrelated pre-transform distribution.
Cache handling must follow accepted sequence state
Drafting and verification both interact with autoregressive state. KV entries associated with rejected speculative suffixes cannot simply become committed target state. The serving implementation needs a way to retain accepted cache positions and discard, overwrite, or avoid committing rejected positions.
This bookkeeping is distinct from the probability rule, but it affects whether the mechanism is practical. Cache layout, block allocation, batching, and rollback cost can determine how much of the theoretical reduction in sequential target invocations becomes observable latency reduction.
Speculative decoding therefore has two separate correctness surfaces. The probability procedure determines whether emitted stochastic samples retain the target distribution. The serving implementation determines whether model state, caches, and sequence positions correspond exactly to the accepted token history. Performance depends on both, but neither can substitute for the other.