Stanford and UC Berkeley Release TokenGate for Learned Token Pruning in Long-Context LLM Prefill
Researchers at Stanford and UC Berkeley have released TokenGate, a drop-in module that adds learned token pruning to pretrained LLMs so long prompts are processed faster. A small MLP gate scores tokens at a chosen layer, and only the top-K are passed to later layers and given KV entries; an MoE-router-style logit bias makes the top-K selection trainable. The team reports 1.4x to 11.3x faster time-to-first-token and 1.5x to 3.4x lower peak prefill memory versus full prefill, tested on Qwen3-1.7B
Why it matters: It gives practitioners an open, trainable approach to cutting long-context prefill cost and memory in existing LLMs with reported multi-fold speedups.