Back to AI News
arxiv

Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure

The paper proposes Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework for LLMs that adapts selection resolution along feature and sequence dimensions. It uses each pretrained head's query-key operator to determine an effective rank for a compact response space, then merges neighboring regions based on directional coherence while keeping heterogeneous regions separate. Exact attention is computed on selected regions using the original full-dimensional KV states. The authors report capturing 88.1% of dense-attention mass, accuracy improvements over UNIQUE of up to 3.47 and 2.57 points on RULER and LongBench-Pro, and up to 14.07x speedup over dense attention.

AI Brief

It offers a concrete training-free method for reducing sparse-attention overhead in long-context LLMs, with reported accuracy and speed comparisons against a recent baseline.

arXiv:2611.09700v1 Announce Type: new Abstract: Attention is a major scalability bottleneck in large language models (LLMs), with its computational cost and key-value (KV) memory traffic becoming increasingly prohibitive as context length grows. Sparse attention addresses this bottleneck by restricting each query to a subset of the context, but its effectiveness hinges on identifying relevant regions both cheaply and reliably. Existing training-free methods primarily rely on runtime relevance estimates, with selection dimensionality and granularity typically governed by fixed heuristics rather than the intrinsic interaction structure of pretrained attention heads. We take a different perspective: the query-key operator of each pretrained head defines a characteristic response structure that determines the resolution needed for sparse selection, both across feature directions and across context tokens. Building on this insight, we propose Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework that jointly adapts selection resolution along these two dimensions. Along the feature dimension, the singular spectrum of each head's query-key operator determines an effective rank, yielding a compact, head-specific response space. Along the sequence dimension, neighboring regions are progressively merged according to the directional coherence of their token responses in this same space, while heterogeneous regions remain separate. Exact attention is then computed using the original full-dimensional KV states of the selected regions. HASA captures 88.1% of dense-attention mass and, compared with UNIQUE, a strong recent sparse-attention baseline, improves accuracy by up to 3.47 and 2.57 points on RULER and LongBench-Pro, respectively, and accelerates self-attention by up to 1.32x, while achieving up to 14.07x speedup over dense attention.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!