Speculative Decoding

Speculative Decoding Depends on Draft Acceptance

Autoregressive generation normally commits one token after each model pass, creating a serial dependency across the output sequence. Speculative decoding changes that execution pattern. A cheaper draft process proposes several future tokens, then the target model evaluates those proposals together and determines which tokens can be committed. The attraction is fewer serial target-model iterations per generated token. That does not make speculative decoding an automatic latency reduction. Its useful operating point depends on how cheaply candidates are produced, how many survive verification, and how much extra work the target model performs while checking them.