arXiv:2611.08303v1 Announce Type: new
Abstract: We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token. Unlike attention mechanisms, where token interactions are governed by positive scalar coefficients, Mamba dynamics involve full matrix-valued interaction coefficients with no positivity or definiteness guarantees. This structural difference makes Mamba dynamics substantially harder to analyze and prevents the direct application of existing techniques for attention-based models. To overcome this challenge, we introduce a new analytical framework based on a high-dimensional invariant box and a sharp exit-time argument, which together recover an effective form of sign-definiteness. Using this framework, we characterize the possible limit points of token trajectories and prove explicit exponential convergence rates. Our analysis reveals that the first token has privileged dynamics: its limiting direction attracts subsequent tokens, inducing token clustering and attention-sink phenomena in deep Mamba models. We further show that this attraction mechanism extends to broader sequence dynamics, including state-space models, causal attention, and classical time-series models. Experiments on Falcon-Mamba-7B validate our predictions, showing that concentration on the first token increases with depth and dominates in later layers.
arxiv
First-Token Attraction in Mamba Dynamics
This paper analyzes token dynamics in deep Mamba models, revealing that the first token acts as an attractor, causing token clustering and attention-sink phenomena. The authors introduce a new analytical framework using a high-dimensional invariant box and sharp exit-time arguments to prove exponential convergence rates. Experiments on Falcon-Mamba-7B confirm that concentration on the first token increases with depth.
Provides a novel theoretical analysis of Mamba architectures with practical validation, offering insights into attention-sink behavior in state-space models.
Comments & Questions (0)
No comments yet
Be the first to comment!