Back to AI News
arxiv

Concentrated Attention Head Dropout Degrades LLM Multi-Step Reasoning More Than Diffuse Noise

An arXiv study evaluates 250 paired GSM8K derivations on Qwen3-8B and LLaMA-3.1-8B-Instruct, comparing clustered attention-head masking with budget-matched diffuse i.i.d. dropout. At onset B=0.10, Qwen3-8B accuracy fell from 91.2% under diffuse noise to 78.8% under burst dropout, while LLaMA-3.1-8B-Instruct dropped from 84.0% to 76.8%. Spreading the same cumulative dropout budget across generation steps produced substantially milder degradation. Error analysis linked Qwen failures to derivation errors and many LLaMA discordant losses to missing final-answer formatting.

AI Brief

It provides concrete evidence that spatially concentrated attention-head disruptions hurt autoregressive reasoning substantially more than diffuse noise, useful for robustness and fault-tolerance work.

arXiv:2611.14209v1 Announce Type: new Abstract: During extended wakefulness, mammalian neocortical networks experience localized periods of transient neuronal silence termed off-states. Here, we examine how analogous spatiotemporally concentrated disruptions affect sequential reasoning in autoregressive Large Language Models (LLMs). Evaluating 250 paired derivations on the GSM8K benchmark across two 8B models (Qwen3-8B and LLaMA-3.1-8B-Instruct), we contrast clustered bursts of attention head masking against budget-matched diffuse i.i.d. dropout. Concentrated bursts induce substantial reasoning degradation: at onset parameter $\mathcal{B}=0.10$ (scheduling $\sim 5.7\%$ of vulnerable head-token volume), Qwen3-8B accuracy drops from $91.2\%$ under diffuse noise to $78.8\%$ under burst dropout ($\Delta = -12.4$ percentage points [pp], exact McNemar $p = 5.5 \times 10^{-6}$), while LLaMA-3.1-8B-Instruct drops from $84.0\%$ to $76.8\%$ ($\Delta = -7.2$ pp, $p = 0.0198$). Conversely, dispersing the identical cumulative budget uniformly across generation steps via stratified quintile matching yields substantially milder degradation (retaining $91.2\%$ in Qwen and $84.0\%$ in LLaMA). In an exploratory factorial dissociation ($N=100$), single-step dropout density is associated with an acute drop in a single schedule draw ($-10.0$ pp on Qwen, $p_{\text{holm}} = 0.0425$), while multi-token continuity exhibits an inconclusive compounding trend ($-7.0$ pp, $p_{\text{holm}} = 0.1892$). Error analysis reveals that Qwen failures reflect derivation errors within formatted responses (conditional accuracy drop of $-11.4$ pp), whereas in LLaMA $22.2\%$ of discordant losses lack a final answer format line (conditional gap narrows to $-5.0$ pp). These findings demonstrate that autoregressive reasoning chains are particularly vulnerable to concentrated structural disruptions, providing empirical benchmarks for fault tolerance in transformer architectures.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!