Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure
The paper proposes Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework for LLMs that adapts selection resolution along feature and sequence dimensions. It uses each pretrained head's query-key operator to determine an effective rank for a compact response space, then merges neighboring regions based on directional coherence while keeping heterogeneous regions separate. Exact attention is computed on selected regions using the original full-dimensional KV states. The authors report capturing 88.1% of dense-attention mass, accuracy improvements over UNIQUE of up to 3.47 and 2.57 points on RULER and LongBench-Pro, and up to 14.07x speedup over dense attention.
It offers a concrete training-free method for reducing sparse-attention overhead in long-context LLMs, with reported accuracy and speed comparisons against a recent baseline.
Comments & Questions (0)
No comments yet
Be the first to comment!