arXiv:2611.08307v1 Announce Type: cross
Abstract: Diffusion language models can write many tokens per step, but must choose which tokens are safe to commit together. Mean-field decoding penalizes pairs of positions whose predictions look alike. We replace this proxy with the coupling itself: how much committing one token changes the model's confidence at another. Cheap probes inside the main forward pass measure it, with no training or extra model, and it plugs into the same mean-field rule. On three models and four benchmarks, our method needs far fewer steps than mean-field decoding and is faster in most settings, also with a key-value cache, at an average accuracy cost of 1.1 points. Shuffling the couplings or replacing them with a uniform bonus loses this gain.
arxiv
Measured Coupling Improves Parallel Decoding in Diffusion Language Models
This paper introduces a method to improve parallel decoding in diffusion language models by directly measuring the coupling between token positions—how committing one token changes confidence at another—using cheap probes inside the main forward pass. This replaces the mean-field decoding proxy that penalizes similar predictions, requiring no training or extra model. On three models and four benchmarks, the method reduces decoding steps and is faster in most settings, including with a key-value cache, at an average accuracy cost of 1.1 points.
Offers a concrete, efficient improvement to diffusion language model decoding with measurable gains, relevant to researchers and practitioners.
Comments & Questions (0)
No comments yet
Be the first to comment!