Large language models generate text autoregressively: each new token depends on the tokens that came before it. That dependency makes ordinary decoding sequential. Even when a GPU has enough compute to process many token positions in parallel, the model normally discovers only one new token per decoding step.

Speculative decoding tries to turn some of that sequential work into parallel verification. A faster draft model proposes several future tokens. The larger target model then scores those proposed positions together and decides which proposals can be accepted. When the draft predicts well, one expensive target-model pass can advance generation by multiple tokens.

The important point is that the draft is a performance helper, not the authority. Correct speculative sampling algorithms use target-model probabilities during verification so that the resulting sampling distribution matches ordinary sampling from the target model, subject to the same numerical implementation details. This article explains that mental model, why acceptance rate matters, and when speculation can make inference slower instead of faster.

Start with the sequential decoding bottleneck

Suppose the current prompt ends with:

The database connection failed because

An autoregressive target model might generate:

the -> server -> rejected -> the -> credentials

Ordinary decoding conceptually does this:

target(prefix)                 -> "the"
target(prefix + "the")         -> "server"
target(prefix + "the server")  -> "rejected"
...

Each step must wait for the previous token because that token becomes part of the next input. A key-value (KV) cache avoids recomputing attention keys and values for the entire prefix, but it does not remove this token-to-token dependency.

This is different from prompt processing, often called prefill. During prefill, the prompt tokens are already known, so the model can process their positions in parallel. During decoding, future tokens are unknown and must normally be discovered one at a time.

Speculative decoding targets this decoding phase.

Use a cheap model to propose, not decide

Assume a small draft model sees the same prefix and proposes four tokens:

the server rejected our credentials

Instead of asking the target model to discover those four tokens in four separate sequential passes, the system feeds the proposed continuation to the target model. Because the candidate tokens are now known inputs, the target can score the relevant positions in parallel in one forward pass.

Conceptually:

draft:
  prefix -> the server rejected our

verification:
  target scores the proposed positions together
          -> accept an initial run of valid proposals

The target does not have to agree with every proposal. Verification proceeds from left to right because later draft tokens were generated assuming that earlier draft tokens were accepted. Once a proposal is rejected, the speculative suffix beyond that point cannot simply be kept as though nothing changed.

This gives speculative decoding its central trade-off:

benefit from verifying several positions together
                    versus
cost of generating proposals that may be rejected

Why verification is more than comparing top tokens

A tempting implementation is:

if draft_token == target_argmax_token:
    accept
else:
    reject

That can be useful in particular deterministic decoding variants, but it is not the general speculative sampling algorithm.

When ordinary generation samples from a probability distribution, preserving the target model’s distribution requires a sampling-aware acceptance and correction procedure. The original speculative-sampling methods use rejection-sampling logic based on the draft and target probabilities. If a draft token is rejected, the replacement token is sampled from a corrected target distribution rather than blindly taking the target’s most likely token.

That distinction matters. A shortcut that merely accepts matching argmax tokens can change the behavior of a sampling configuration. If exact distributional equivalence is a requirement, use a speculative-decoding implementation whose verification rule provides that property for the chosen sampling setup; do not infer it from the phrase “speculative decoding” alone.

The broader mental model is stable across implementations:

  1. a cheap mechanism proposes candidate tokens;
  2. the target evaluates those candidates;
  3. verification determines the accepted prefix;
  4. generation resumes from the first position that still needs a target-consistent token.

Measure accepted progress, not just draft speed

A very fast draft model is not automatically a useful draft model. It must also predict tokens that the target is likely to accept.

Suppose the draft proposes four tokens per round. Compare two simplified cases:

case A: accept 4, reject 0
case B: accept 1, reject the next proposal

In case A, the target verification pass advances several token positions. In case B, the system paid for draft generation but gained little extra progress before returning to another round.

This is why acceptance rate and accepted tokens per verification step are important operational metrics. They connect draft quality to actual useful work. The exact metric names vary by serving system, so define them explicitly when comparing deployments.

Draft-target agreement depends on more than model size. It can change with:

  • the prompt and workload;
  • the draft and target tokenization requirements of the implementation;
  • decoding parameters such as temperature;
  • how many tokens are proposed per round;
  • domain mismatch between the draft and target models.

A draft that works well for repetitive code completion may behave differently for open-ended prose.

Choosing the speculation length is a balancing problem

Let the draft propose k tokens in each round. Increasing k creates more opportunity to advance several positions with one target verification, but it also creates more speculative work that can be wasted after an early rejection.

For example:

k = 2
accepted: [A, B]
useful draft work: 2 tokens

k = 8
accepted: [A, B]
rejected at token 3
unused speculative suffix: tokens 4 through 8

The second configuration offered a larger theoretical gain, but the early rejection made much of the draft work unnecessary. Conversely, setting k too low can leave available verification parallelism unused.

There is therefore no universal best speculation length. Tune it against the real workload and hardware. Some systems adapt the number of proposed tokens dynamically; others use a fixed value because simpler scheduling is easier to operate.

Understand where the speedup comes from

Speculative decoding does not reduce the mathematical capability required from the target model. It changes how target-model work is scheduled.

The technique is attractive when scoring several known candidate positions in a target pass costs much less than discovering those positions through the same number of separate target decoding steps. Modern accelerators can often exploit the additional parallel work, while ordinary single-token decoding may be constrained by repeatedly reading model weights and KV-cache data for small amounts of new computation.

But the target still performs verification work, and the draft adds its own compute and memory traffic. Real latency improvement therefore depends on the complete serving path, not just the number of accepted tokens.

A useful approximate way to reason about one speculative round is:

round time ≈ draft proposal time + target verification time + coordination overhead

The round is worthwhile only if the accepted progress makes this cheaper than producing the same progress with ordinary target decoding.

Do not turn that expression into a universal speedup formula. Draft and verification work can overlap in some systems, hardware utilization is nonlinear, batching changes costs, and rejected proposals alter the amount of progress per round. Benchmark end-to-end latency instead.

Account for memory and serving trade-offs

A second model is not free. Keeping a draft model resident may consume accelerator memory that could otherwise hold a larger KV cache, more concurrent requests, or a larger batch. Moving the draft to another device can introduce communication overhead.

This creates a deployment trade-off between latency and capacity. A configuration that improves single-request decoding latency can still reduce total serving throughput if it consumes enough memory or disrupts batching.

Measure at least:

  • time to first token separately from time per output token;
  • end-to-end request latency at realistic output lengths;
  • accepted tokens per verification round;
  • draft and target accelerator utilization;
  • memory used by model weights and KV caches;
  • throughput under the expected concurrency level.

Speculative decoding primarily changes token generation after prefill, so it may have little effect on time to first token when prompt processing dominates that metric.

Common mistakes that erase the benefit

Choosing the smallest possible draft model

A tiny draft can be cheap per token but disagree with the target so often that verification rarely advances far. Optimize the combined system, not draft latency in isolation.

Choosing a draft that is too expensive

A highly accurate draft can also fail economically if producing proposals costs nearly as much as ordinary target decoding. Agreement must compensate for proposal cost.

Assuming every speculative implementation is lossless

Specific speculative sampling algorithms can preserve the target distribution. Other techniques described as speculative decoding may use different verification rules, approximations, tokenizers, or decoding restrictions. Treat equivalence as an algorithmic guarantee that must be documented and tested, not as a property of the name.

Benchmarking only easy prompts

Highly predictable continuations can make acceptance look excellent. Include the prompt types, temperatures, output lengths, and concurrency levels that production actually sees.

Ignoring batching effects

Serving many requests together already creates parallel work for the accelerator. Speculation changes sequence progress unevenly because different requests accept different numbers of draft tokens. That can complicate batching and scheduling. A large single-request improvement does not guarantee the same gain under high concurrency.

When speculative decoding is a good fit

Consider it when autoregressive decoding latency is important, the target model is expensive enough that avoiding sequential target steps has meaningful value, and a substantially cheaper proposal mechanism can achieve useful agreement with the target.

It is especially worth evaluating for workloads with long generated outputs, because decoding occupies a larger fraction of total request time. Predictable domains can also help if they allow a cheap draft to propose accepted continuations frequently.

A simpler setup may be better when prompts are long but outputs are very short, prefill dominates latency, request batching already saturates the hardware efficiently, memory is tight, or no cheap draft produces enough accepted tokens. In those cases, KV-cache management, batching, quantization, or a smaller target model may address the actual bottleneck more directly.

Validate the optimization against the baseline

Treat ordinary target-model decoding as the reference implementation. Before deploying speculation, run the same representative workload through both paths.

For deterministic configurations, compare generated outputs where the implementation promises equivalent behavior. For sampling configurations, verify the documented distribution-preservation guarantees and test that your integration uses compatible sampling settings. In both cases, compare latency and throughput under production-like load.

Also test failure boundaries: very short outputs, low draft acceptance, long contexts, high concurrency, and memory pressure. An optimization that only wins on the median easy request can be a poor default if its tail behavior is costly.

Conclusion

Speculative decoding accelerates autoregressive generation by separating proposal from authority. A cheap draft predicts several possible future tokens, while the target model verifies those positions and remains responsible for target-consistent generation.

The technique works when parallel verification saves more expensive sequential target steps than the draft and coordination costs add. That makes draft quality, draft cost, speculation length, batching, memory, and workload shape part of the same engineering decision. Measure accepted progress and end-to-end serving performance together; neither draft speed nor acceptance rate alone tells you whether speculation is actually helping.

Artificial Intelligence LLM Inference Performance