Concentrated Attention Head Dropout Degrades LLM Multi-Step Reasoning More Than Diffuse Noise
An arXiv study evaluates 250 paired GSM8K derivations on Qwen3-8B and LLaMA-3.1-8B-Instruct, comparing clustered attention-head masking with budget-matched diffuse i.i.d. dropout. At onset B=0.10, Qwen3-8B accuracy fell...