Spectral Degeneration of Self-Attention under Isotropic Score Geometry
This paper introduces an idealized spectral model for the pre-softmax score matrix in transformers, motivated by the rotational isotropy of query and key representations. The authors prove that the row-wise softmax map t...