Research
Search Order Shapes Language Models' Default Solutions on Constraint Puzzles
A new arXiv study examines four open-weight models from three families on symbolic puzzles and multi-solution ZebraLogic variants. It finds repeated samples concentrate on a single default solution far beyond uniform sampling, and different models often converge on the same answer. Prompts that specify a search order—such as visiting entities with the fewest initial candidates first—shift responses toward the favored solution by 26–40 percentage points. Randomizing presentation or supplying a random search order increases solution breadth by 26–42 points on symbolic tasks and 11–26 on ZebraLogic, with validity losses of at most 3–4 points.
LLM reasoningPrompt designOutput diversity