Back to AI News
arxiv

Maya a1: Do the Options Read the Model, or Does Their Position? Order Sensitivity in a Non-Autoregressive Decision Model

Researchers re-engineered a 421M-parameter non-autoregressive multiple-choice checkpoint and tested whether its answer depends on where an option is listed. Across 951 Qwen-verified questions under cyclic rotations, the released checkpoint changed answers for 25.8% of questions, or 19.7% on the 605 English-language items, with a 10.1-point accuracy gap between gold-first and gold-last placement. Fine-tuning with option shuffling reduced the flip rate to 7.2%, versus 11.4% for identical rows without shuffling, while a confidence-weighted vote over orderings reached 69.7% versus 69.1% for a single pass at 4.1x compute. Fine-tuning with the authors' parallel layout on the same rows produced a 0.5% flip rate.

AI Brief

It quantifies a concrete robustness failure in option-scoring models and shows that option shuffling during fine-tuning can partly mitigate it.

arXiv:2611.09803v1 Announce Type: new Abstract: Laya answers typed multiple-choice questions in one forward pass by scoring a hidden state at one [MASK] marker per option. We re-engineer the checkpoint (421M parameters; a 2-layer option head with no positional tensor; a documented but unused parallel layout) and test whether the answer depends on where an option is listed. Presenting 951 Qwen-verified questions under every cyclic rotation of their options (or six random permutations for 12 options), the released checkpoint changes its answer for 25.8% [22.9-28.7] of questions (19.7% on the 605 English-language items alone); accuracy with the gold option first versus last differs by 10.1 points, and the flips concentrate where the model's own confidence is low (54% versus 8% across confidence terciles). Fine-tuning with Laya's trainer on 2,818 teacher-verified rows with option shuffling cuts the flip rate to 7.2%; the identical rows without shuffling give 11.4%, so shuffling is responsible for 4.2 points of the improvement. A confidence-weighted vote over orderings gives 69.7% against 69.1% for a single base pass at 4.1x the compute. Labels are LLM-made and the test sets are teacher-filtered. We also fine-tune with the authors' parallel layout on the same rows: flip rate 0.5%.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!