Autoregressive generation normally advances one token at a time: the model scores the next-token distribution, a decoding rule selects a token, and that token becomes part of the context for the next forward step. Speculative decoding changes the execution schedule. A cheaper draft process proposes several future tokens, while the target model retains authority over which proposals can enter the generated sequence.
That separation is the core constraint. Draft tokens are predictions about future target-model decisions, not a replacement distribution that can be appended unchecked. The serving implementation must verify them against target-model scores and apply the acceptance rule required by the chosen speculative algorithm.
Drafting converts serial guesses into a verification batch
Assume the accepted prefix is x and the draft process proposes:
d1, d2, d3, d4The target model can evaluate the extended sequence and produce next-token distributions for the relevant positions in one verification pass:
p1(. | x)
p2(. | x, d1)
p3(. | x, d1, d2)
p4(. | x, d1, d2, d3)The positions are still causally ordered. Batching their verification does not make token generation non-autoregressive; it changes when target-model computation is scheduled. Each target distribution at a later position is conditioned on earlier draft tokens supplied in that verification input.
If an earlier draft token is rejected, later draft positions that depend on it cannot simply be committed as though the rejected prefix had been accepted. The algorithm stops or repairs the speculative span according to its defined rule, then resumes from the resulting accepted prefix.
Acceptance semantics depend on the decoding algorithm
For greedy decoding, verification can be conceptually simple. A draft token at a position is accepted when it matches the token the target model would select under the same greedy rule. Verification proceeds through the draft span until the first mismatch. A target-selected token can then replace the rejected proposal, subject to the exact implementation.
Sampling requires stricter treatment. Comparing a draft token only with the target model’s argmax does not preserve the target sampling distribution. Speculative sampling algorithms instead define acceptance probabilities from draft and target probabilities and, on rejection, draw from a correction distribution. The exact equations belong to the selected algorithm; replacing them with an informal match test changes the output distribution.
This distinction also prevents a common implementation shortcut: speculative decoding is not defined merely by running a small model first and checking whether a large model considers its tokens plausible. A probability threshold, top-k membership test, or logit-distance rule creates a different decoder unless that rule is part of a specified algorithm with known semantics.
The target model remains the distributional authority
A useful implementation invariant is that draft confidence does not grant acceptance by itself. A draft model can assign high probability to a token that the target model scores differently. The verifier must use target-model outputs for the acceptance decision required by the algorithm.
This matters when the draft process is not a conventional smaller model. Draft proposals can come from another head, a truncated computation path, cached structures, or another mechanism. Those designs can change cost and proposal quality, but the same boundary remains: proposal generation and target verification are separate roles.
The draft process also does not need to reproduce the full target distribution perfectly to be useful. Its proposals need enough agreement with target decisions for speculative work to avoid being discarded too often. The useful operating point depends on draft cost, accepted-span length, target verification cost, batch shape, hardware, and serving implementation. No acceptance rate alone establishes a universal speed gain.
Rejection shortens useful speculative work
Suppose the first two proposals are accepted and the third is rejected:
draft: d1 d2 d3 d4
status: ok ok reject unusedThe fourth proposal was computed from a prefix containing d3. Once d3 is not part of the accepted sequence, d4 is not a continuation of the committed prefix. Its draft computation may have consumed resources, but it cannot be retained as an independently verified future token under the original causal path.
This creates a practical tension in speculative span length. Longer spans offer more candidate positions per verification pass when agreement persists. They also expose more draft work to invalidation after an early rejection. The suitable span therefore depends on workload and implementation rather than on a fixed architectural constant.
KV cache state must follow committed tokens
Speculative execution also affects cache management. Draft and target execution can temporarily create key-value state for tokens that are not ultimately committed. A serving engine must distinguish provisional cache entries from state corresponding to the accepted sequence.
After rejection, target-model cache state must represent the actual committed prefix before ordinary or speculative decoding continues. Implementations may accomplish this with rollback metadata, page-level cache management, selective commit, recomputation, or other internal mechanisms. The representation is implementation-specific; the semantic requirement is not. Rejected speculative tokens must not silently remain as causal context for subsequent target generation.
This boundary becomes more visible under continuous batching, paged KV caches, and request preemption. Scheduler bookkeeping, sequence length, cache ownership, and accepted-token count have to describe the same committed prefix even when provisional work exists internally.
Throughput and latency gains are conditional
Speculative decoding reduces the number of serial target-model decoding iterations only when enough proposed tokens survive verification and the verification work maps favorably to the serving hardware. It also adds draft computation, acceptance logic, provisional state, and sometimes extra memory traffic.
As a result, a configuration that helps one serving regime can be neutral or unfavorable in another. Short outputs may leave little room to amortize setup and draft work. Heavy batching can change the relative cost of target verification. A draft mechanism with weak agreement can spend substantial compute on tokens that are discarded. Kernel support and cache layout can also alter the balance.
The precise implementation boundary is therefore narrower than the phrase “generate several tokens at once” suggests. Speculative decoding creates provisional future tokens, verifies them under target-model semantics, commits only the accepted prefix, and keeps cache state aligned with that commitment. Any optimization around the mechanism has to preserve those four properties before its performance characteristics matter.