Autoregressive language models generate text one token at a time. Even when an accelerator has substantial parallel compute available, the model normally cannot determine token 12 until token 11 is known. That dependency makes generation latency difficult to reduce simply by adding more parallel hardware.
Speculative decoding attacks this bottleneck by doing cheap work ahead of the expensive model. A faster draft model proposes several future tokens. The full target model then evaluates those proposals together and accepts the portion that is consistent with its own distribution. With the appropriate acceptance-and-correction algorithm, this changes how generation is computed without changing the distribution that the target model defines.
This article develops the idea from a small example, explains why verification can save sequential target-model steps, and shows the practical conditions that determine whether speculation actually improves latency.
Start with the sequential decoding bottleneck
Suppose a target language model has generated:
The database connectionStandard autoregressive decoding conceptually proceeds like this:
prefix -> target model -> next token: "failed"
prefix + "failed" -> target model -> next token: "because"
prefix + "failed because" -> target model -> next token: "the"
...Each next-token decision depends on the preceding generated token. Producing four new tokens therefore requires four sequential decoding steps, even though the numerical work inside each step is highly parallel.
This serial dependency is different from ordinary request batching. Batching lets a server process work from multiple sequences together. It does not remove the token-by-token dependency inside one sequence.
Speculative decoding tries to make one expensive target-model verification advance the sequence by more than one token.
The mental model: propose cheaply, verify expensively
Assume a smaller draft model sees the same prefix and quickly proposes four tokens:
prefix: The database connection
draft: failed because the passwordInstead of asking the target model for these four tokens one at a time, the serving system evaluates the proposed continuation with the target model in a verification pass.
Imagine the target model agrees with the first three draft positions but not the fourth:
draft: failed | because | the | password
verify: yes | yes | yes | rejectThe system can keep the accepted prefix:
The database connection failed because theand then continue from the rejection point according to the decoding algorithm.
The important unit is the accepted prefix. Once a speculative token is rejected, later draft tokens were conditioned on a continuation that is no longer valid, so a simple linear speculative round cannot keep them merely because they look plausible in isolation.
If several draft tokens are accepted, one target-model verification has replaced several sequential target-model decoding steps. If rejection happens immediately, much of the speculative work was wasted.
Why the target can verify several positions together
During training and prompt processing, transformers routinely compute predictions for many positions in parallel while a causal attention mask prevents each position from seeing future tokens.
Verification can use the same property. Given the prefix plus a proposed sequence, the target model can evaluate the conditional distributions needed for multiple draft positions in one forward pass over those proposed positions. Causal masking preserves the correct dependency structure: the score for a proposed token depends only on the prefix and earlier tokens, not later proposals.
This does not mean autoregressive generation has become fully parallel. The proposals still need to come from somewhere, and rejected proposals create another sequential round. Speculation is useful because the drafter is cheaper than the target and because target verification can amortize expensive work across multiple proposed positions.
Greedy verification is the simplest teaching case
For deterministic greedy decoding, the basic idea is especially easy to see. The target model’s highest-probability token at each verified position can be compared with the corresponding draft token.
Suppose the drafter proposes:
A B C Dand greedy target verification implies:
A B X ...Then A and B match the target’s greedy continuation. C is the first mismatch, so the speculative prefix ends there and the target continuation uses X instead.
This simplified case teaches the central mechanism, but production systems often use stochastic sampling. For sampling, merely accepting a draft token whenever it equals one independently sampled target token is not the general lossless algorithm.
Exact speculative sampling needs acceptance and correction
Let the draft model assign probability q(x) to candidate token x, while the target model assigns p(x) at the same position. A standard speculative-sampling construction accepts a proposed token with probability:
min(1, p(x) / q(x))When p(x) >= q(x), the proposal can be accepted. When the draft places more probability on the token than the target does, acceptance is reduced so the final output does not overrepresent that token.
If a proposal is rejected, the replacement is not generally sampled from the original target distribution p without adjustment. The correction distribution is proportional to the positive residual:
max(0, p(x) - q(x))normalized across the vocabulary.
This correction is what compensates for probability mass already represented by the draft proposal process. Together with the acceptance rule, it allows the resulting token distribution to match the target model’s sampling distribution under the algorithm’s assumptions.
The practical lesson is important: speculative decoding is not just “small model guesses, large model approves.” Distribution-preserving sampling depends on a specific verification procedure. A serving implementation should use a tested speculative-decoding algorithm rather than inventing an intuitive accept/reject shortcut.
Draft quality and draft cost determine the useful speedup
A good drafter needs two properties that pull in different directions:
- It must be cheap enough that proposing tokens costs much less than generating them with the target model.
- Its predictions must be close enough to the target that several proposed tokens are accepted per round.
A tiny but inaccurate drafter may produce proposals quickly but cause frequent early rejection. A larger drafter may achieve a higher acceptance rate while consuming enough compute that the extra work erases the latency benefit.
A useful mental model for one speculative round is:
round cost ~= draft cost + target verification cost
round benefit ~= number of useful tokens advancedThe exact timing is hardware- and implementation-dependent. Verification of several positions is not free, and its cost does not necessarily equal one ordinary single-token decoding step. Memory bandwidth, kernel shapes, KV-cache operations, batch size, and draft length all affect the result.
For that reason, acceptance rate alone is not a sufficient performance metric. Measure end-to-end latency and throughput on the workload that matters.
Draft length creates another trade-off
The draft length is the number of tokens proposed before target verification.
A longer draft offers more potential progress per successful round:
2-token draft -> at most a short accepted run
8-token draft -> potentially a much longer accepted runBut later proposals are useful only if all earlier proposals in the linear draft survive. When disagreement tends to occur early, generating many additional draft tokens wastes work and makes verification larger.
A fixed draft length can be reasonable as a starting point. More advanced serving systems may adapt how much they speculate based on recent acceptance behavior or other signals. Such policies are implementation choices rather than guarantees of speculative decoding itself.
Tune draft length against measured latency. A value that works for code completion may not work for open-ended dialogue because predictability, sequence lengths, and sampling settings differ.
Sampling settings can change acceptance behavior
The target distribution used during verification must correspond to the decoding distribution you actually intend to sample from. Temperature and probability truncation methods such as top-k or nucleus sampling can change that distribution, so the speculative implementation must apply compatible processing at the correct stage.
The drafter and target also need a well-defined relationship between their token spaces for the straightforward token-level algorithm. Many common implementations use models with compatible tokenization because proposed token IDs can then be evaluated directly by the target. Methods for heterogeneous vocabularies exist, but they require additional machinery and should not be assumed to behave like the simple shared-token case.
Even with compatible vocabularies, a draft model from an unrelated training setup may predict very different continuations. Compatibility is therefore more than matching tensor shapes: useful speculation requires enough distributional agreement to offset drafting overhead.
Measure the quantities that explain performance
When evaluating speculative decoding, record more than requests per second. Useful measurements include:
- End-to-end generation latency: whether users actually receive completed output sooner.
- Inter-token latency: whether visible streaming becomes smoother or more bursty.
- Accepted tokens per verification round: how much sequential target work each round replaces.
- Acceptance by draft position: whether later speculative positions are routinely wasted.
- Draft and target execution time: whether the drafter is consuming the expected fraction of latency.
- Throughput under concurrency: whether gains for one request survive realistic server load.
- Memory use: whether keeping a second model and its runtime state reduces feasible batch size or creates other pressure.
Compare against the same target model using the same intended decoding policy without speculation. Otherwise a quality or sampling change can be mistaken for an inference optimization.
Common mistakes
Treating draft tokens as authoritative
The drafter exists to propose cheap work, not to replace the target’s decision rule. Accepting proposals based on an arbitrary confidence threshold can change the generated distribution and therefore the model behavior.
If exact target-distribution preservation matters, use a verification algorithm designed to provide it.
Choosing the drafter only by parameter count
Parameter count is an imperfect proxy for serving cost. Architecture, precision, hardware utilization, memory movement, and implementation all affect latency. Benchmark the actual pair on the intended hardware.
Optimizing acceptance rate in isolation
A more expensive drafter can improve acceptance while making total generation slower. The objective is useful tokens advanced per unit of end-to-end time, not the highest acceptance percentage.
Assuming every workload benefits equally
Highly predictable continuations can be easier to speculate than uncertain ones. Long prompts, short outputs, large batches, different sampling temperatures, and memory-constrained deployments can all shift the cost balance.
Speculative decoding should therefore be treated as a workload-dependent optimization rather than a universal property of LLM serving.
When speculative decoding is a good fit
Consider it when single-request or low-concurrency generation latency matters, target-model decoding is expensive, and a substantially cheaper drafter can predict the target well enough to produce useful accepted runs. It is particularly worth testing when the serving stack already supports an exact or otherwise well-characterized verification method.
A simpler decoding path may be preferable when outputs are very short, the target model is already small, accelerator utilization is dominated by large concurrent batches, memory cannot comfortably hold the additional drafter, or measured acceptance is too low to repay the extra work.
It is also worth separating speculative decoding from other optimizations. Quantization can reduce model memory and arithmetic cost. Batching can improve aggregate hardware utilization. KV caching avoids recomputing attention keys and values for the established prefix. Speculative decoding addresses a different problem: the serial sequence of target-model decoding decisions. These techniques can interact, but one does not automatically replace another.
Conclusion
Speculative decoding reduces the cost of autoregressive serialization by moving some prediction work to a cheaper proposer and letting the target verify multiple proposed positions together. Its value comes from advancing several valid tokens per expensive target round, not from trusting a smaller model’s guesses.
The core engineering trade-off is simple: drafting must be cheap, agreement must be high enough, and verification must amortize target-model work. Preserve the intended target distribution with a correct acceptance procedure, then tune draft length and model pairing with end-to-end measurements. When those conditions hold, speculative decoding can reduce generation latency without requiring the target model itself to predict future tokens independently.