Re-Search probes LLM research ideation with a Twenty Questions-style game
The paper introduces Re-Search, an evaluation that adapts Twenty Questions to research ideation: a model is given only a published paper's research problem and must rediscover its core idea through yes-or-no questions. Built from 200 ICML 2026 papers, it tracks how a model's hypothesis evolves under binary oracle feedback, isolating ideation from execution. Across six recent LLMs, nearly all progress occurred within the first 30 of 100 questions, after which most models plateaued. Controlled interventions indicate the bottleneck is generation rather than judgment: models can recognize good ideas or questions when offered but often cannot produce them independently.
It offers a concrete evaluation method for LLM research ideation and identifies a generation bottleneck with direct implications for designing autoresearch systems.
Comments & Questions (0)
No comments yet
Be the first to comment!