Search Order Shapes Language Models' Default Solutions on Constraint Puzzles
A new arXiv study examines four open-weight models from three families on symbolic puzzles and multi-solution ZebraLogic variants. It finds repeated samples concentrate on a single default solution far beyond uniform sampling, and different models often converge on the same answer. Prompts that specify a search order—such as visiting entities with the fewest initial candidates first—shift responses toward the favored solution by 26–40 percentage points. Randomizing presentation or supplying a random search order increases solution breadth by 26–42 points on symbolic tasks and 11–26 on ZebraLogic, with validity losses of at most 3–4 points.
It gives concrete evidence that prompt-level search order, not just sampling, controls which correct answer a language model defaults to, with practical implications for prompt design and output diversity.
Comments & Questions (0)
No comments yet
Be the first to comment!