Back to AI News
arxiv

Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer

A new arXiv study finds that storing cached recurrent state in bf16 with round-to-nearest makes hybrid reasoning models more verbose and increases 32K-token cap hits. Across six open hybrid models and 590 competition-math problems, five of six reasoned 1.07x to 1.27x longer; Qwen3.5-9B accuracy fell from 85.6% to 70.8% and cap hits rose from 118 to 223. On LiveCodeBench, the two tested models lost 12.6 and 13.4 pass@1 points, and a 128K budget only halved the accuracy gap. Stochastic rounding in bf16 largely removes the effect, with an integer-hash generator matching FP32 accuracy and cap hits at bf16 memory and decoding speed.

AI Brief

It gives inference and evaluation teams a concrete numerical-precision failure mode in hybrid reasoning models plus a practical mitigation.

arXiv:2611.12127v1 Announce Type: cross Abstract: Hybrid reasoning models interleave attention with recurrent mixers such as Mamba-2 and Gated DeltaNet, whose state is cached between decoding steps. Prior work showed that storing this state in FP16 or INT8 with round-to-nearest (RTN) makes Mamba-2 and GDN hybrids more verbose, and that stochastic rounding (SR) fixes it. We study the other 16-bit format that serving engines expose, bf16, on six open hybrid reasoning models and 590 competition-math problems. With bf16 RTN state, five of six models reason longer (geometric-mean length 1.07x to 1.27x; also under greedy decoding in the two models tested) and hit the 32K token cap more often. On Qwen3.5-9B, cap hits rise from 118 to 223 and accuracy falls from 85.6% to 70.8%. On LiveCodeBench, both models tested also reason longer and lose 12.6 and 13.4 points of pass@1. Rerunning the capped items with a 128K budget halves the accuracy gap but does not close it (93.6% vs. 86.6%): bf16 runs that never finish are twice as common, and 85% of them are exact repetition loops. On Nemotron-Nano-9B-v2, by contrast, 30 of 31 extra caps finish at 128K, so the cost there is mostly extra tokens (exploratory). Same-width bf16 SR removes the length increase in four of five affected models, restores Qwen3.5-9B to 87.6% and brings code length and pass@1 near FP32 levels, so the failure is mostly rounding bias, not missing resolution. With an integer-hash random generator, SR on Qwen3.5-9B matches FP32 on cap hits and accuracy (113 vs. 118; 85.6% for both) at the memory and decoding speed of bf16 RTN. In the Nemotron-Nano-v2 lineage, SR on 25% of heads (an intermediate-decay band) removes the effect, while random heads and the top teacher-forced-KL octiles do not; in Nemotron-3 and Qwen3.5-9B the effect is spread across heads. We recommend that evaluations of hybrid models report the state dtype, cap rate and output length alongside accuracy.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!