Back to AI News
arxiv

Two-Clustering Regime of Token Dynamics in Causal Attention

The paper partially resolves and extends a conjecture on clustering in causal attention masking to time-dependent parameter regimes. It proves that two-cluster convergence holds when the largest eigenvalue of the value transformation is positive and is generally false when it is non-positive. The work identifies a dimension-reduction phenomenon in which token trajectories collapse onto a two-dimensional structure, and introduces an 'instability within the flow' framework for analyzing non-monotone, non-autonomous attention dynamics. Simulations and experiments use Group Query Attention layers.

AI Brief

It offers theoretical grounding for why token representations cluster in causal attention and a dimension-reduction lens useful for analyzing transformer dynamics.

arXiv:2611.18088v1 Announce Type: new Abstract: We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue $\lambda_{\max}$ of the value transformation. We prove that this behavior holds when $\lambda_{\max}>0$ and is generally false when $\lambda_{\max}\le0$. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework--instability within the flow--inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!