Research
New Study Reveals How Irrelevant Context Degrades LLM Accuracy and How to Fix It
A new arXiv paper identifies a mechanism called the Intrusion Pathway that explains why language models lose accuracy when prompts contain irrelevant, off-topic context. The authors localize the problem to a sparse set of 'intrusion heads' and show that knocking out about 1% of attention heads recovers 57% of the performance drop at bounded cost. This offers a practical intervention for improving robustness in RAG, tool-using, and long-context systems.
language model robustnessattention mechanismsretrieval-augmented generation