Back to AI News
arxiv

Search Order Shapes Language Models' Default Solutions on Constraint Puzzles

A new arXiv study examines four open-weight models from three families on symbolic puzzles and multi-solution ZebraLogic variants. It finds repeated samples concentrate on a single default solution far beyond uniform sampling, and different models often converge on the same answer. Prompts that specify a search order—such as visiting entities with the fewest initial candidates first—shift responses toward the favored solution by 26–40 percentage points. Randomizing presentation or supplying a random search order increases solution breadth by 26–42 points on symbolic tasks and 11–26 on ZebraLogic, with validity losses of at most 3–4 points.

AI Brief

It gives concrete evidence that prompt-level search order, not just sampling, controls which correct answer a language model defaults to, with practical implications for prompt design and output diversity.

arXiv:2611.25916v1 Announce Type: cross Abstract: When a constraint puzzle has several correct answers, language models tend to give the same one.We study four open-weight models from three families on symbolic puzzles and multi-solution ZebraLogic variants with exactly known solution sets.Repeated samples concentrate on one default solution far beyond uniform sampling, and different models often share it.Yet models usually produce even rarely sampled solutions when added clues uniquely identify them.On held-out symbolic puzzles, we compare visiting entities in name order with visiting those with the fewest initial candidates first. Both searches try values in the stated order. Where their pre-specified predictions differ, the latter matches a larger share of responses in all four models, by 26 to 40 percentage points.Reversing the stated value order or specifying a search order shifts responses towards the solution that the new order favours, on both testbeds. Randomising the presentation or supplying a random search order raises breadth (the effective share of solutions in use) by 26 to 42 percentage points on the symbolic task and 11 to 26 on ZebraLogic. The largest observed validity losses are 3 and 4 percentage points, respectively. Randomised presentation broadens answers more than a random-choice instruction in each of the three main models.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!