Back to AI News
arxiv

Query-Guided RoPE Modulation Targets Lost-in-the-Middle in Long-Context LLMs

Researchers propose QGPM, a training-free method that reshapes rotary position embeddings using query relevance derived from the model's own attention heads. It builds a broad relevance map for lower layers and a sharp one for upper layers, advancing low-relevance tokens by less than one position on fast-rotating planes while keeping slow-rotating planes in their original layout. Experiments on NoLiMa and RULER across multiple context lengths show improvements over base models and strong baselines, with the largest gains on retrieval tasks, and the method generalizes across several backbones. Code is publicly available.

AI Brief

Readers get a concrete, training-free technique for improving long-context retrieval without retraining, backed by experiments across multiple backbones.

arXiv:2611.09554v1 Announce Type: cross Abstract: Long-context language models often neglect information in the middle of the input, a phenomenon known as "lost-in-the-middle". Part of this gap may stem from the long-range decay of rotary position embedding (RoPE), which weakens the positional signal between distant tokens. Existing input compression deletes low-importance tokens, and attention reweighting acts only after RoPE has weakened this signal. RoPE-level methods modify positions directly but apply the same remapping to every query. To address these limitations, we propose QGPM (Query-Guided Positional Modulation), a training-free method that reshapes RoPE positions using query relevance from the model's own attention heads. QGPM builds a broad relevance map for lower layers and a sharp one for upper layers. Low-relevance tokens advance by less than one position on fast-rotating planes, pulling relevant evidence toward the query. Slow-rotating planes keep the original layout, with every token in place. Experiments on NoLiMa and RULER with multiple context lengths show that QGPM improves over base models and strong baselines, with the largest gains on retrieval tasks, and generalizes across multiple backbones. Our code is available at https://github.com/iZHpy/QGPM.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!