First-Token Attraction in Mamba Dynamics
This paper analyzes token dynamics in deep Mamba models, revealing that the first token acts as an attractor, causing token clustering and attention-sink phenomena. The authors introduce a new analytical framework using...