Live intelligence

Latest AI updates

1 updates
Research

First-Token Attraction in Mamba Dynamics

This paper analyzes token dynamics in deep Mamba models, revealing that the first token acts as an attractor, causing token clustering and attention-sink phenomena. The authors introduce a new analytical framework using a high-dimensional invariant box and sharp exit-time arguments to prove exponential convergence rates. Experiments on Falcon-Mamba-7B confirm that concentration on the first token increases with depth.

Mamba architecturetoken clusteringattention sink