speculative-decoding
1 articles
Apprentice Scribe Hedgehog at the Oracle Tower: Sneaking in Answers Made the Prophecies Several Times Faster, and the Oracle Didn't Lose a Single Word. Speculative Decoding 101
Every time the clockwork Oracle in the Oracle Tower spits out one word, the entire tome has to be hauled up from the bottom of the tower. Mogu, the apprentice scribe hedgehog in the mushroom hat, gets bored enough to start writing answers in advance. The prophecies get several times faster, and not one word changes. This light-novel-style tutorial teaches speculative decoding: why LLM generation is stuck on memory bandwidth, how a small model guesses ahead, how the big model checks a whole row at once, and the acceptance rule that leaves the output probability distribution exactly unchanged.