Research
First-Token Attraction in Mamba Dynamics
This paper analyzes token dynamics in deep Mamba models, revealing that the first token acts as an attractor, causing token clustering and attention-sink phenomena. The authors introduce a new analytical framework using a high-dimensional invariant box and sharp exit-time arguments to prove exponential convergence rates. Experiments on Falcon-Mamba-7B confirm that concentration on the first token increases with depth.
Mamba architecturetoken clusteringattention sink