Latent Sparse Attention: Subquadratic Attention for Ultra-Long Contexts
Latent Sparse Attention (LSA) is a new attention method that clusters keys hierarchically in a learned latent space, allowing queries to score clusters rather than individual keys. This reduces computational cost from qu...