Speculative Decoding: How LLM Inference Cheats the Autoregressive Bottleneck
TL;DR — Autoregressive LLM inference is memory-bandwidth-bound, not compute-bound — the GPU sits mostly idle waiting for weight reads. Speculative decoding breaks that bottleneck by drafting multiple candidate tokens cheaply and verifying them in a single parallel forward pass. This essay…