Autoregressive generation normally commits tokens one position at a time. Even when a large model has ample parallel compute available, each next-token decision depends on the prefix produced so far. That serial dependency makes decoding latency sensitive to the number of target-model passes.
Speculative decoding changes the unit of work. A cheaper draft model proposes several candidate tokens, then the target model evaluates the proposed continuation in one pass. Accepted candidates advance generation by multiple positions without requiring one separate target pass per accepted token.
The technique does not make the target model trust the draft model. Verification remains the gate that determines which proposed tokens can enter the output.
Drafting moves some serial work to a cheaper model
Assume the current prefix is x. A draft model extends it with a short candidate block:
x -> d1 -> d2 -> d3 -> d4The draft still generates those candidates autoregressively, so its own work is serial. The intended gain comes from making that serial work inexpensive relative to running the target model once per token.
The target model can score positions across the candidate block in parallel because the full proposed sequence is already available as input. Verification then determines how far the candidate prefix can be accepted before generation must account for a disagreement.
This distinction matters when estimating cost. Speculative decoding adds draft computation and verification logic. It pays off only when those additions cost less than the target-model work they replace.
Acceptance controls how much progress each verification pass makes
A candidate block has value only to the extent that verification accepts useful progress from it. If the target model commonly agrees with the draft over several consecutive positions, one target pass can advance generation by several tokens. If disagreement occurs near the start of most blocks, the draft has spent computation on candidates that contribute little progress.
Acceptance is therefore tied to the relationship between the two model distributions, not merely to draft-model size. A very small draft model may be cheap but disagree often. A larger draft model may produce candidates closer to the target while consuming more time itself.
The useful operating point depends on both effects:
net gain = target work avoided - draft and verification overheadThis expression is conceptual rather than a universal latency formula. Actual runtime also depends on hardware utilization, kernel behavior, cache handling, sequence length, and implementation details.
Candidate-block length creates a related tension. Longer blocks provide more potential progress per target pass, but later candidates are useful only if earlier positions survive verification. Repeated early rejection can make long drafts wasteful.
Exact sampling needs a correction rule
Greedy generation has a direct notion of agreement: a proposed token can be accepted when it matches the token selected by the target model under the same decoding rule.
Sampling requires more care. Simply accepting draft samples that look plausible and falling back to target samples after rejection can alter the output distribution. Standard speculative sampling uses an acceptance and correction procedure constructed so that the resulting samples follow the target model’s distribution.
That distribution-preserving property belongs to the decoding algorithm, not to every system that happens to use a draft model. An implementation that changes the acceptance test, mixes model probabilities, truncates distributions differently, or applies incompatible sampling transforms can produce a different distribution.
This boundary is useful in system design. A draft model can be much less capable than the target without directly defining output quality, provided the verification procedure preserves the target decoding distribution. Draft quality then affects computational cost through acceptance rather than serving as the final authority on generated tokens.
Tokenization affects the verification contract
The simplest draft-target pairing uses a compatible tokenization scheme. Candidate positions then have a direct token-level correspondence between the models.
Different tokenizers complicate that correspondence because the same text can split into different token sequences. A draft token may map to part of a target token, several target tokens, or a segmentation with different boundaries. Implementations that support such pairs need an explicit re-encoding and alignment mechanism before target verification.
This is not merely an input-format detail. Verification operates over model token positions, so token alignment becomes part of the correctness path. A system should not assume that two causal language models can participate in token-level speculation solely because both can decode the same text.
Cache behavior can erase part of the expected gain
Autoregressive inference commonly stores key and value states from prior positions so each new pass does not recompute the entire prefix. Speculative decoding adds state for draft generation and for target verification over candidate positions.
The exact cache strategy is implementation-specific. A rejected suffix may require cache entries to be discarded or adjusted, while accepted positions need to become part of the committed prefix. Extra copying, allocation, synchronization, or cache conversion can reduce the latency saved by fewer target passes.
Memory capacity matters as well. Running a separate draft model may require additional weights and its own cache. A draft model that is computationally cheap can still be unattractive when the deployment is constrained by device memory.
Self-speculative variants address a different point in this design space by deriving draft predictions from part of the target model. Such methods can share more model state, but they require architecture or checkpoint support for the chosen early-exit mechanism. They are not interchangeable with arbitrary two-model speculation.
Throughput and single-request latency are separate measurements
Speculative decoding is often motivated by reducing serial target-model calls for one sequence. That does not imply the same gain under every serving workload.
A server with many concurrent requests can batch target-model work across sequences. In that setting, hardware that was underused by single-sequence decoding may already be kept busy. Draft execution, variable acceptance lengths, and verification can also complicate scheduling.
For this reason, acceptance rate alone is not a sufficient performance metric. A useful evaluation records end-to-end latency, generated tokens per unit time, target-model passes, draft cost, memory use, and the request concurrency that produced those measurements.
Comparisons also need the same output contract. If one configuration preserves the target sampling distribution and another intentionally changes it, a latency comparison is no longer isolating the effect of speculative execution.
The draft model is a performance component, not an output authority
Speculative decoding is most effective when candidate generation is cheap, agreement is frequent enough to advance several positions per target verification, and the surrounding runtime can exploit parallel scoring of the proposed block. None of those conditions follows from model parameter count alone.
The target model still defines the decoding distribution under the standard exact algorithm. The draft model changes how candidate work is scheduled ahead of verification. Treating draft choice, block length, cache policy, and acceptance statistics as inference-system parameters keeps the central distinction clear: speculation can reduce serial target work, but only verification determines which speculative work becomes committed output.