Study compares weight quantization against option-order averaging under a fixed inference budget
The paper asks whether lower-bit weight storage degrades the benefit of averaging answer-option reorderings, and which use of a fixed budget is cheaper. Using simulated round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B, evaluated on ARC-Easy, ARC-Challenge, MMLU and HellaSwag with 300 items each, the authors report that 6-bit weights lost no accuracy against fp16 while 4-bit raised answer-slot skew. Averaging four cyclic orders recovered +12.6 points at 4 bits for the 0.6B model. In cost-matched pairs measured as weight bytes read times passes, extra orders beat extra bits only at 6 or more bits, leading to a suggested rule of 6-bit weights plus up to four order passes.
It offers practitioners a concrete, cost-matched decision rule for splitting a fixed inference budget between weight precision and option-order averaging in small local models.
Comments & Questions (0)
No comments yet
Be the first to comment!