Live intelligence

Latest AI updates

3 updates
Research

Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure

The paper proposes Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework for LLMs that adapts selection resolution along feature and sequence dimensions. It uses each pretrained head's query-key operator to determine an effective rank for a compact response space, then merges neighboring regions based on directional coherence while keeping heterogeneous regions separate. Exact attention is computed on selected regions using the original full-dimensional KV states. The authors report capturing 88.1% of dense-attention mass, accuracy improvements over UNIQUE of up to 3.47 and 2.57 points on RULER and LongBench-Pro, and up to 14.07x speedup over dense attention.

sparse attentionLLM inference efficiencylong-context LLMs
Industry

Alibaba RTPurboV2: 10x Sparse Attention & the Resurgence of Native Transformer

Alibaba RTPurboV2: The Renaissance of Native Transformer with 10x Sparse AttentionThe Resurgence of Full AttentionAs the demand for long sequences driven by widespread Agent applications grows, the Attention mechanism in traditional GPT architectures<!--autointro-->

RTPurboV2Sparse AttentionFull Attention