Autoregressive generation normally advances one accepted token at a time. Each new token extends the prefix, so the next target-model evaluation depends on the token selected at the preceding position. Speculative decoding changes the execution schedule: a cheaper draft process proposes several future tokens, then the target model evaluates those positions together and decides how much of the proposal can be retained.
That rearrangement can reduce the number of serial target-model calls per emitted token. It does not make verification free, and a longer draft block is not automatically better. The useful operating point depends on how quickly proposals are produced, how often they survive target verification, and what the serving stack spends on rejected work.
Draft tokens are proposals, not committed output
Consider a prefix x and a draft model that proposes a block:
d1, d2, d3, d4The target model evaluates conditional distributions for the corresponding positions. Verification then processes the proposed tokens in order. Once a proposal is rejected, later draft tokens were generated from a prefix that is no longer the accepted path, so they cannot simply remain committed.
This prefix dependency is the central implementation boundary. A four-token draft does not guarantee four accepted tokens. If the first three survive and the fourth fails, the cycle advances by the accepted prefix plus whatever token the algorithm selects at the rejection boundary. If an early token fails, more draft work becomes unusable.
The exact acceptance and replacement rules depend on the speculative algorithm. For sampling schemes designed to reproduce the target distribution, rejection requires a correction rule rather than an arbitrary resample from the unmodified target distribution. Greedy variants have different semantics. A serving system therefore cannot treat every feature called speculative decoding as interchangeable.
Acceptance converts parallel verification into forward progress
A target verification pass can score multiple proposed positions in one model invocation because the candidate prefix is already available for that pass. The amount of useful progress, however, is determined by the accepted prefix length.
Suppose a draft block contains K tokens. If all K proposals survive, one verification cycle can commit several positions before another serial target step is needed. If rejection often occurs near the front of the block, the same verification width produces much less progress.
This creates a distinction between verification width and accepted-token throughput. Increasing K expands the amount of candidate work available to verify, but it also exposes more positions at which the draft and target can diverge. The later proposals are useful only when every earlier proposal needed to reach them has survived.
An aggregate acceptance percentage can also hide position effects. A system with frequent rejection at the first proposal behaves differently from one that usually accepts a long prefix and rejects near the end, even when a coarse acceptance statistic looks similar. For serving analysis, accepted prefix length per cycle is often closer to the execution mechanism being optimized.
A more accurate draft can still be a poor serving choice
Proposal quality is only one side of the cost equation. The draft path must be cheaper enough to justify the target work it avoids.
A larger draft model may produce proposals that agree with the target more often, yet take longer to produce each proposal. A smaller draft may be much cheaper but diverge sooner. Neither property alone determines end-to-end latency.
The relevant comparison includes at least the cost of generating the draft block, target verification, acceptance processing, cache updates, and any recovery work after rejection. Runtime scheduling also matters when requests are batched: extra draft execution can contend for accelerator capacity that could otherwise serve target-model work.
For that reason, acceptance rate should not be reported as if it were a direct speedup metric. It describes proposal survival under a particular workload and decoding configuration. Latency and throughput remain system measurements shaped by model sizes, sequence lengths, batching, kernels, memory traffic, and scheduler policy.
Longer draft blocks increase both opportunity and wasted work
Draft length controls how far the speculative path runs before target verification. A short block limits the maximum progress per cycle but also limits proposal work that can be discarded after an early mismatch. A long block creates more opportunity to amortize a target invocation when agreement remains high.
The cost becomes asymmetric after rejection. If proposal d2 fails in an eight-token block, tokens d3 through d8 were computed from a speculative continuation that is no longer valid for the accepted sequence. Their computation may have helped neither output progress nor the next accepted prefix.
Adaptive implementations can vary draft length based on runtime signals, but the policy itself becomes part of serving behavior. A fixed K, a confidence-based stop rule, and a history-based controller can produce different verification shapes under the same prompt and models. Comparisons need to state which policy produced the measurements.
Sampling settings can change acceptance behavior
Draft-target agreement is conditional on the decoding process, not only on the model pair. Temperature, truncation filters, token constraints, and application-specific logit processing can alter the distributions used during proposal and verification.
For distribution-preserving speculative sampling, the draft and target probabilities participate directly in acceptance and correction. Changing decoding settings can therefore change rejection frequency even when model weights remain fixed. A benchmark collected under greedy decoding does not establish the same acceptance behavior under stochastic sampling.
Tokenizer compatibility is another hard boundary for common token-level designs. Verification operates on proposed token identities at specific positions. If draft and target paths do not share the required token representation, the implementation needs an explicit mapping or a different speculative method; token IDs cannot be assumed to correspond merely because decoded text can look similar.
KV cache handling must follow the accepted prefix
Both draft and target execution commonly use KV caches to avoid recomputing the full prefix. Speculation creates cache state for positions that may later be rejected.
After verification, cache state must correspond to the sequence that actually survives. Retaining KV entries for rejected continuation tokens would make subsequent attention operate on a prefix different from the committed output. Implementations can avoid, truncate, overwrite, or stage speculative cache entries in different ways, but the logical boundary is the same: future decoding state must match the accepted token history.
This also affects memory planning. A serving runtime may need temporary capacity for speculative positions beyond the currently committed sequence. The amount depends on block length, batch shape, cache layout, and implementation strategy, so speculative decoding should not be modeled as a pure compute optimization with no state-management cost.
Evaluation needs cycle-level signals
Useful telemetry connects proposal work to accepted progress. Draft tokens produced, accepted prefix length, rejection position, verification time, draft time, and emitted tokens per cycle expose different parts of the mechanism. End-to-end request latency and throughput remain necessary because local improvements can be offset elsewhere in the serving path.
Metrics should also be segmented by conditions that materially change agreement, such as model pair, decoding configuration, prompt or sequence regime, and draft-length policy. A single global acceptance number can conceal the cases that dominate cost.
The practical boundary is simple: speculative decoding exchanges extra candidate computation for fewer serial target-model steps. Its value comes from enough accepted forward progress to repay proposal, verification, and state-management overhead. The draft model is useful not merely when it predicts plausible text, but when its proposals survive the target algorithm often enough at a low enough execution cost for the serving workload in question.