Live intelligence

Latest AI updates

1 updates
Research

Concentrated Attention Head Dropout Degrades LLM Multi-Step Reasoning More Than Diffuse Noise

An arXiv study evaluates 250 paired GSM8K derivations on Qwen3-8B and LLaMA-3.1-8B-Instruct, comparing clustered attention-head masking with budget-matched diffuse i.i.d. dropout. At onset B=0.10, Qwen3-8B accuracy fell from 91.2% under diffuse noise to 78.8% under burst dropout, while LLaMA-3.1-8B-Instruct dropped from 84.0% to 76.8%. Spreading the same cumulative dropout budget across generation steps produced substantially milder degradation. Error analysis linked Qwen failures to derivation errors and many LLaMA discordant losses to missing final-answer formatting.

LLM reasoning robustnessattention head dropouttransformer fault tolerance