Research
Two-Clustering Regime of Token Dynamics in Causal Attention
The paper partially resolves and extends a conjecture on clustering in causal attention masking to time-dependent parameter regimes. It proves that two-cluster convergence holds when the largest eigenvalue of the value transformation is positive and is generally false when it is non-positive. The work identifies a dimension-reduction phenomenon in which token trajectories collapse onto a two-dimensional structure, and introduces an 'instability within the flow' framework for analyzing non-monotone, non-autonomous attention dynamics. Simulations and experiments use Group Query Attention layers.
transformer theoryattention dynamicsrepresentation clustering