Speculative Decoding Trades Draft Accuracy for Target Model Work
Autoregressive generation normally commits tokens one position at a time. Even when a large model has ample parallel compute available, each next-token decision depends on the prefix produced so far. That serial dependency makes decoding latency sensitive to the number of target-model passes. Speculative decoding changes the unit of work. A cheaper draft model proposes several candidate tokens, then the target model evaluates the proposed continuation in one pass. Accepted candidates advance generation by multiple positions without requiring one separate target pass per accepted token.