Live intelligence

Latest AI updates

2 updates
Research

Query-Guided RoPE Modulation Targets Lost-in-the-Middle in Long-Context LLMs

Researchers propose QGPM, a training-free method that reshapes rotary position embeddings using query relevance derived from the model's own attention heads. It builds a broad relevance map for lower layers and a sharp one for upper layers, advancing low-relevance tokens by less than one position on fast-rotating planes while keeping slow-rotating planes in their original layout. Experiments on NoLiMa and RULER across multiple context lengths show improvements over base models and strong baselines, with the largest gains on retrieval tasks, and the method generalizes across several backbones. Code is publicly available.

long-context LLMsRoPEattention
Research

Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure

The paper proposes Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework for LLMs that adapts selection resolution along feature and sequence dimensions. It uses each pretrained head's query-key operator to determine an effective rank for a compact response space, then merges neighboring regions based on directional coherence while keeping heterogeneous regions separate. Exact attention is computed on selected regions using the original full-dimensional KV states. The authors report capturing 88.1% of dense-attention mass, accuracy improvements over UNIQUE of up to 3.47 and 2.57 points on RULER and LongBench-Pro, and up to 14.07x speedup over dense attention.

sparse attentionLLM inference efficiencylong-context LLMs