Autoregressive generation normally commits one token after each model pass, creating a serial dependency across the output sequence. Speculative decoding changes that execution pattern. A cheaper draft process proposes several future tokens, then the target model evaluates those proposals together and determines which tokens can be committed.
The attraction is fewer serial target-model iterations per generated token. That does not make speculative decoding an automatic latency reduction. Its useful operating point depends on how cheaply candidates are produced, how many survive verification, and how much extra work the target model performs while checking them.
Drafting moves work ahead of commitment
Consider a target model that has already committed a prefix. Instead of asking it for only the next token, a draft model extends that prefix with a candidate block:
committed prefix
|
+-- draft -> d1 -> d2 -> d3 -> d4
|
+-- target verifies candidate positionsThe target can process the candidate sequence in a form that exposes distributions for several positions during one verification pass. A valid speculative algorithm then commits an accepted prefix and obtains the next token according to its verification rule.
This changes the unit of progress. Ordinary decoding usually advances one committed token per serial target invocation. A speculative iteration can advance several tokens when the candidate prefix agrees sufficiently with the target distribution.
The draft model is not authoritative. Its role is to propose computation that may save target-model iterations. A cheap draft with poor agreement can spend little time proposing tokens but trigger frequent rejection. A more capable draft can raise acceptance while consuming more of the latency budget itself.
Exact sampling requires more than matching top tokens
There are multiple algorithms described as speculative decoding, and their output guarantees are not interchangeable.
For stochastic generation, the original speculative sampling construction uses acceptance and correction rules based on both draft and target token distributions. Under its stated assumptions, the resulting samples follow the target model distribution rather than the draft distribution. The verifier is doing more than checking whether both models selected the same top token.
A simplified greedy variant has a different condition. Candidate tokens can be compared with the target model’s greedy choices, accepting the matching prefix and replacing the first mismatch with the target choice. That can reproduce ordinary greedy decoding when the same target computation and token-selection rules are used.
These cases should not be collapsed into a generic claim that verification preserves output. Distribution-preserving sampling depends on the specific acceptance and correction algorithm. Greedy equivalence depends on deterministic selection and consistent model state. Relaxed verification methods may intentionally accept a different output distribution in exchange for other system goals.
Acceptance length controls useful progress
Suppose a draft proposes k tokens. The target verifies the block, but only the first a draft tokens survive before a rejection, where 0 <= a <= k. The iteration’s value depends heavily on a.
If rejection often occurs at the first position, the system repeatedly pays drafting and block-verification costs while making little speculative progress. If long prefixes are accepted, one target verification can replace several serial target invocations.
Acceptance is sequential even when verification computation is parallelized. Once a candidate token is rejected, later draft tokens were conditioned on a prefix that will not be committed. They cannot simply be retained as though the rejected token had remained in the sequence.
This makes early candidate quality especially consequential. An error near the start of a draft block invalidates more downstream speculative work than an error near its end.
Longer draft blocks expose a cost boundary
Increasing the number of proposed tokens creates two opposing effects.
A longer block offers more potential committed tokens from one verification pass. It also asks the draft to predict farther into a future that has not yet been confirmed by the target. Any early rejection leaves the unused suffix as wasted draft work, and verification still processes candidate positions that may not become output.
The useful block size therefore depends on the local acceptance pattern and the relative costs of draft and target execution. A fixed block length can be suitable for one workload and inefficient for another.
Dynamic lookahead methods adjust the number of proposed tokens instead of treating it as a constant. The general motivation follows directly from the cost structure: when confidence or recent acceptance suggests that additional proposals are likely to survive, extending the block can create more useful work; when rejection risk rises, stopping drafting earlier can avoid computation on a suffix with low expected value.
The policy itself adds another decision mechanism, so its benefit still has to be measured in the complete serving path.
Verification is parallel, not free
Speculative decoding is often described in terms of replacing several target passes with one. That description can hide the shape of the verification pass.
The target model evaluates multiple candidate positions. Even when accelerator parallelism makes that operation much cheaper than the same number of serial decoding steps, it performs more token-position work than a single-token target step. Attention state, kernel shapes, memory traffic, and serving-engine implementation affect the resulting cost.
A rough latency model for one speculative iteration can be written as:
T_iteration = T_draft(k) + T_verify(k) + T_overheadThe relevant quantity is not T_iteration alone but the committed tokens produced by that iteration. A useful comparison is therefore time per committed token under the same request conditions.
This also explains how a high acceptance rate can coexist with weak end-to-end improvement. If drafting is expensive, verification scales poorly with block length, or orchestration overhead is large, accepted tokens may not compensate for the added work.
Batch pressure changes the economics
A decoding technique that reduces latency for one request does not necessarily increase throughput under a saturated serving workload.
Target verification introduces variable block lengths and variable accepted lengths. Different requests can reject at different positions. A scheduler may need to combine speculative verification with continuous batching, cache allocation, and request admission. The resulting accelerator utilization can differ from isolated single-request measurements.
Draft execution also consumes compute and memory resources. If the draft and target share an accelerator, draft work competes with target work. If they run on separate devices, transfer and coordination enter the critical path. A draft that appears inexpensive from model size alone can still be poorly placed in the serving topology.
For this reason, speculative decoding should be evaluated against the actual objective: time to first token, inter-token latency, request completion latency, throughput at a target concurrency, or some combination. One metric cannot stand in for the others.
Tokenization and model compatibility constrain the draft
The simplest draft-target arrangement uses compatible token semantics so candidate token IDs have a clear interpretation for verification. A smaller model from the same model family is a common fit, but model size alone does not guarantee useful agreement.
The draft also needs access to the committed context required to produce its proposals. Its cache state and positional handling must remain consistent as accepted tokens advance the sequence and rejected suffixes are discarded.
Alternative speculative methods can generate candidates without a conventional smaller language model, including approaches based on additional prediction heads or token patterns already present in context. Those designs change drafting cost and compatibility constraints, but the same core accounting remains: proposal work is valuable only when verification converts enough of it into committed target-model output.
Measure rejection as a systems signal
Acceptance rate is informative, but a single aggregate percentage can hide where time is lost. Two workloads can accept the same fraction of draft tokens while producing different latency if one rejects mostly at the first candidate and the other rejects near the end of long blocks.
Useful instrumentation includes accepted prefix length, proposed block length, draft time, verification time, and committed tokens per iteration. These measurements can be grouped by request type or sequence region when acceptance behavior changes across a workload.
Comparisons also need matched decoding settings. Changing temperature, sampling rules, stopping conditions, or target-model configuration at the same time as the speculative path makes attribution difficult.
Speculative decoding is most useful when candidate generation is cheap relative to target decoding and the verifier can convert candidate blocks into multiple committed tokens often enough to offset rejected work. The mechanism reduces a serial dependency; it does not remove the cost of predicting tokens. The practical boundary is set by how much speculative computation becomes valid target-model progress.