Maya a1: Do the Options Read the Model, or Does Their Position? Order Sensitivity in a Non-Autoregressive Decision Model
Researchers re-engineered a 421M-parameter non-autoregressive multiple-choice checkpoint and tested whether its answer depends on where an option is listed. Across 951 Qwen-verified questions under cyclic rotations, the released checkpoint changed answers for 25.8% of questions, or 19.7% on the 605 English-language items, with a 10.1-point accuracy gap between gold-first and gold-last placement. Fine-tuning with option shuffling reduced the flip rate to 7.2%, versus 11.4% for identical rows without shuffling, while a confidence-weighted vote over orderings reached 69.7% versus 69.1% for a single pass at 4.1x compute. Fine-tuning with the authors' parallel layout on the same rows produced a 0.5% flip rate.
It quantifies a concrete robustness failure in option-scoring models and shows that option shuffling during fine-tuning can partly mitigate it.
Comments & Questions (0)
No comments yet
Be the first to comment!