Autoregressive language models generate text sequentially. After processing the prompt, the model predicts a next token, appends that token to the sequence, and repeats the process. That dependency makes generation difficult to parallelize across time: token 101 cannot normally be generated until token 100 is known.

Speculative decoding changes the amount of useful work performed during each expensive target-model step. A cheaper draft process proposes several future tokens, then the target model verifies those proposals together. When enough proposals are accepted, the application can advance by multiple tokens while invoking the large model fewer times.

The important idea is not simply “use a small model to generate text.” The target model still determines the final output distribution. The draft exists to predict work that the target can verify efficiently.

Start with the sequential generation bottleneck

Suppose a model has already processed this prompt:

The database connection failed because

A normal autoregressive decoder might generate:

the -> server -> was -> unavailable

Conceptually, each arrow requires another decoding step:

prompt -> target -> "the"
       -> target -> "server"
       -> target -> "was"
       -> target -> "unavailable"

Modern inference engines reuse cached attention state, so they do not recompute the entire prefix from scratch at every step. Even with that optimization, generation still has a serial dependency between tokens.

For an interactive application, this often appears as inter-token latency: the delay between one generated token and the next.

The draft-and-verify mental model

Speculative decoding introduces a cheaper way to guess several upcoming tokens.

Imagine that a draft model proposes:

the server was unavailable

Instead of trusting those tokens directly, the target model evaluates the proposed continuation. A speculative decoding algorithm then accepts a valid prefix of the proposal and handles the first disagreement according to its acceptance rule.

The flow is roughly:

draft proposes:  the server was unavailable
                         |
                         v
target verifies: token probabilities for the proposal
                         |
                         v
result:           accept a prefix, then continue

If several proposed tokens are accepted, one target verification step can move generation forward by several positions. If the proposal is poor, fewer tokens are accepted and much of the draft work provides little benefit.

That leads to the central trade-off:

Speculative decoding is useful when drafting and verifying several likely tokens costs less than generating those tokens one target-model step at a time.

Verification is different from trusting the draft

A naive system could ask a small model to generate several tokens and append them without checking. That would be faster, but it would change which model is producing the answer.

Speculative decoding instead uses an acceptance procedure designed around the target model’s probabilities. In the original sampling formulation, rejected draft proposals are corrected using a residual distribution so that the resulting samples follow the target model’s distribution rather than the draft model’s distribution.

This distinction matters. A correctly implemented lossless speculative sampling algorithm is an inference optimization: it changes how samples are produced, not the probability distribution that the target model defines.

Implementations can also use related speculative techniques with different guarantees. When evaluating an inference engine, check whether its method is lossless with respect to the target distribution or intentionally trades some output fidelity for additional speed.

Why the target can verify multiple positions together

At first, it may seem that the target model must still process each proposed token one by one.

The difference is that the draft has already supplied a candidate sequence. The target therefore knows the proposed tokens for several positions and can evaluate them in a batched forward pass, subject to the normal causal attention mask. Each position can attend only to the prefix available at that position, but accelerator hardware can process computations for multiple positions in parallel.

This is valuable because large-model decoding can be limited by repeatedly moving model parameters and performing relatively small amounts of work for one new token. Verifying a short block can make better use of the same target-model invocation.

The exact performance depends on the model, hardware, inference engine, batch size, sequence length, and speculative implementation. Speculation does not remove the computational cost of verification.

Acceptance rate determines how much speculation pays off

Consider a draft that proposes four tokens per round.

If the target regularly accepts all four, generation may advance several positions for each target verification. If the target usually rejects the first proposal, the system pays for drafting and block verification while advancing very little.

A useful measurement is therefore the number of accepted speculative tokens per verification round. Acceptance rate is related, but the practical question is how much target-model progress each round produces.

High agreement is more likely when the draft predicts the target well. Agreement can fall when:

  • the draft and target have substantially different behavior;
  • the text is difficult or highly uncertain;
  • sampling settings encourage a wider range of tokens;
  • the proposed block is so long that errors accumulate later in the proposal.

This is why “propose more tokens” is not automatically faster. A longer speculative block creates more opportunity for useful acceptance, but it also spends more draft work and can include a larger rejected suffix.

The draft model must be cheap and compatible

For model-based speculation, the draft model should be substantially cheaper to run than the target while still predicting the target’s likely continuations well enough to earn useful acceptance.

A draft that is almost as expensive as the target leaves little room for savings. A tiny but poorly aligned draft can also perform badly because proposals are rejected too often.

Compatibility matters as well. Practical model-based implementations commonly require tokenization that lets proposed tokens be interpreted consistently by the target. The exact constraints depend on the inference engine and speculative algorithm, so a seemingly related smaller model is not automatically a valid draft.

The right pair is therefore a systems choice, not merely a parameter-count choice.

A simple latency model

A rough mental model helps explain when the technique can win.

Without speculation, generating k tokens costs approximately k target decoding steps:

cost ~= k * target_step

With speculation, one round adds draft cost and target verification cost:

round_cost ~= draft_proposal + target_verification

The round is beneficial only if the accepted progress justifies that combined cost.

For example, if a round accepts four tokens but drafting and verification together cost almost as much as four ordinary target steps, the latency improvement will be small. If those four accepted tokens can be produced for substantially less cost, speculation can reduce generation time.

This is intentionally not a universal formula. Real engines overlap work, batch requests, use different kernels, and pay costs that change with sequence length. Measure end-to-end behavior on the workload you actually serve.

Measure latency separately from throughput

A common mistake is to evaluate speculative decoding using only tokens per second aggregated across a server.

Interactive systems often care about at least three different quantities:

  • time to first token, which includes prompt processing and scheduling before generation begins;
  • inter-token latency, which describes how quickly subsequent output arrives;
  • throughput, which describes total useful generation across requests over time.

Speculative decoding primarily targets the serial cost of generating subsequent tokens. It does not inherently make prompt prefill faster.

It can also interact with server batching. Extra draft work and larger verification blocks consume compute and memory that might otherwise serve another request. A configuration that improves single-request latency is not guaranteed to maximize throughput under heavy concurrency.

Benchmark both the latency experienced by individual requests and the capacity of the whole serving system.

Compare speculation with other inference optimizations

Speculative decoding addresses a different bottleneck from several common LLM optimizations.

KV caching avoids recomputing attention keys and values for the existing prefix during autoregressive generation. It is a foundational decoding optimization and is useful whether or not speculation is enabled.

Quantization reduces numeric precision for some model data or operations, often reducing memory use and potentially improving performance on suitable hardware. It changes the cost of running the model rather than the serial dependency between generated tokens.

Continuous batching schedules token-generation work from multiple requests together to improve accelerator utilization. It is primarily a serving-throughput technique, although scheduling decisions also affect latency.

Speculative decoding tries to make more than one token of progress per expensive target verification round.

These techniques can coexist. The best combination depends on whether the deployment is constrained by latency, throughput, memory, or a mixture of them.

Know when speculative decoding may not help

Speculation adds moving parts, so it should solve a measured problem.

It may provide limited value when responses are very short. There may not be enough generated tokens to recover the overhead of drafting and verification.

It can also be unattractive when the target model is already small and cheap. Adding a second model or speculative component can increase memory use and operational complexity without producing meaningful latency savings.

Low draft acceptance is another warning sign. If most speculative work is discarded, ordinary decoding may be simpler and competitive.

Finally, a high-throughput server can have different priorities from a single interactive request. Additional speculative work can compete with batching efficiency, so benchmark under realistic concurrency rather than assuming a single-stream speedup transfers directly to production load.

Avoid common implementation mistakes

Treating the draft output as authoritative

The draft proposes candidates; it does not replace target verification in a lossless speculative algorithm. Appending draft tokens directly is a different generation strategy.

Measuring only draft acceptance

A high acceptance rate is encouraging, but it does not prove an end-to-end speedup. Draft execution, verification, scheduling, and memory overhead still matter.

Making the speculative block as long as possible

Longer proposals can increase accepted progress, but later tokens are more likely to depend on an earlier disagreement. Tune proposal length using measured latency rather than intuition alone.

Assuming every speculative implementation has identical guarantees

“Speculative decoding” describes a family of techniques. Some preserve the target distribution; related methods may approximate it. Verify the guarantee provided by the implementation you deploy.

Benchmarking an unrealistic workload

A single warm request does not represent a production server with concurrent users, varying prompt lengths, and different output lengths. Measure the traffic pattern that matters to the application.

A practical evaluation process

When deciding whether to deploy speculative decoding, begin with an ordinary target-model baseline. Record time to first token, inter-token latency, end-to-end latency, throughput, and memory use under representative load.

Then enable one speculative configuration and measure:

accepted speculative tokens per round
end-to-end request latency
inter-token latency
server throughput
memory usage

Change one major variable at a time, such as draft model or proposal length. This makes it easier to connect a performance change to its cause.

For applications where output distribution must remain unchanged, also confirm that the implementation provides the required lossless sampling guarantee for the decoding mode you use. Performance is not a substitute for semantic correctness.

Use speculation when serial decoding is the real bottleneck

Speculative decoding is easiest to understand as prediction plus verification. A cheap process predicts several future tokens; the expensive target checks those predictions in a form that can exploit more parallel computation; accepted predictions let generation advance faster.

Its effectiveness depends on balance. The draft must be cheap enough, its proposals must agree with the target often enough, and verification must be efficient enough for accepted progress to outweigh the added work.

That makes speculative decoding a strong option for latency-sensitive LLM serving, but not a switch that should be enabled blindly. Establish a baseline, measure accepted progress and end-to-end performance, and keep the optimization only when it improves the workload users actually experience.

Artificial Intelligence LLM Inference Performance