Query-Guided RoPE Modulation Targets Lost-in-the-Middle in Long-Context LLMs
Researchers propose QGPM, a training-free method that reshapes rotary position embeddings using query relevance derived from the model's own attention heads. It builds a broad relevance map for lower layers and a sharp one for upper layers, advancing low-relevance tokens by less than one position on fast-rotating planes while keeping slow-rotating planes in their original layout. Experiments on NoLiMa and RULER across multiple context lengths show improvements over base models and strong baselines, with the largest gains on retrieval tasks, and the method generalizes across several backbones. Code is publicly available.