Back to AI News
arxiv

Study compares weight quantization against option-order averaging under a fixed inference budget

The paper asks whether lower-bit weight storage degrades the benefit of averaging answer-option reorderings, and which use of a fixed budget is cheaper. Using simulated round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B, evaluated on ARC-Easy, ARC-Challenge, MMLU and HellaSwag with 300 items each, the authors report that 6-bit weights lost no accuracy against fp16 while 4-bit raised answer-slot skew. Averaging four cyclic orders recovered +12.6 points at 4 bits for the 0.6B model. In cost-matched pairs measured as weight bytes read times passes, extra orders beat extra bits only at 6 or more bits, leading to a suggested rule of 6-bit weights plus up to four order passes.

AI Brief

It offers practitioners a concrete, cost-matched decision rule for splitting a fixed inference budget between weight precision and option-order averaging in small local models.

arXiv:2611.14413v1 Announce Type: new Abstract: Local language models can save memory by storing weights in fewer bits, and they can reduce sensitivity to the order of answer options by averaging a few reorderings. We ask whether the first makes the second worse, and which is the cheaper way to spend a fixed budget. On Qwen3-0.6B and Qwen3-1.7B with simulated round-to-nearest weight quantization (8, 6, 4 and 3 bits, fp16 reference), read from the logits of the letters A to D on ARC-Easy, ARC-Challenge, MMLU and HellaSwag (300 items each), 6 bits lose no accuracy against fp16 (they score +1.8 (+1.0 to +2.5) and +0.9 (+0.3 to +1.5) points higher, which we do not explain), and 8 bits change it by -0.7 (-1.1 to -0.3) and -0.4 (-0.7 to -0.2) points. At 4 bits the answer-slot skew rises from 0.15 to 0.53 (0.6B) and from 0.07 to 0.12 (1.7B); Gaussian noise at the same weight SNR gives 0.15 and 0.37 (two draws) and 0.19 (one draw), so the control is inconclusive on whether rounding adds slot skew beyond equal-power noise. Flip rates rise but stay below an accuracy-matched baseline of independent uniform errors. Averaging four cyclic orders recovers +12.6 points at 4 bits for 0.6B (+5.3 at fp16). We price bits and passes in one machine-independent unit, weight bytes read times passes. In cost-matched pairs, extra orders beat extra bits only when the weights have 6 or more bits, and the decision table reduces to: use 6-bit weights and spend the rest of the budget on up to four orders, one pass each. Everything is simulated on two small models with the simplest quantizer; a title-level search found no earlier study of quantization together with option order.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!