Phrase-Level Speculative Decoding Drafts Multi-Word Units on a Frozen Target
The paper proposes changing the drafting unit for speculative decoding from single target tokens to one- to three-word phrases stored in a separate draft vocabulary. A small MLP head reads the frozen target model's hidden state, predicts a phrase entry, and a greedy longest-match expansion maps it back into the target tokenizer's tokens; candidates are verified in one forward pass behind a prefix-tree mask. Because the target and tokenizer stay frozen and candidates are accepted only when they match the target's argmax, the authors state the served output is identical to greedy decoding up to floating-point tie-breaking. On a single A5000 at batch 1 against HuggingFace generate(), a 21.94M-parameter head reports 2.391x on GSM8K, 2.444x on MATH-500 and 1.544x on MT-Bench, with a smaller 8,000-entry head described as 64x smaller than EAGLE-3's drafter.
It offers a concrete, parameter-efficient alternative to token-level speculative drafting with reported throughput gains and an unchanged greedy output guarantee, useful for readers working on LLM inference acceleration.
Comments & Questions (0)
No comments yet
Be the first to comment!