Two-Clustering Regime of Token Dynamics in Causal Attention
The paper partially resolves and extends a conjecture on clustering in causal attention masking to time-dependent parameter regimes. It proves that two-cluster convergence holds when the largest eigenvalue of the value transformation is positive and is generally false when it is non-positive. The work identifies a dimension-reduction phenomenon in which token trajectories collapse onto a two-dimensional structure, and introduces an 'instability within the flow' framework for analyzing non-monotone, non-autonomous attention dynamics. Simulations and experiments use Group Query Attention layers.
It offers theoretical grounding for why token representations cluster in causal attention and a dimension-reduction lens useful for analyzing transformer dynamics.
Comments & Questions (0)
No comments yet
Be the first to comment!