Alignment by Confident Debate: Preference Learning from Multi-Agent Debate
AbCD converts multi-agent debate traces into preference data without gold-answer labels. It clusters final answers by task-level semantic equivalence and combines answer equivalence with confidence-weighted disagreement. Tested on mathematical reasoning, multiple-choice quantitative QA, and executable code generation, post-training on debate-derived preferences outperformed inference-only ensembling, debate, and SFT on the same traces across five model families from 1B to 8B parameters and four datasets. The work reports a maximum consensus-accuracy gain of 21.5 percentage points and also examines limitations of heterogeneous committees and agreement-based weighting.
It offers a concrete self-supervised preference-learning method for turning multi-agent debate into training supervision, with controlled evaluations across reasoning and code tasks.
Comments & Questions (0)
No comments yet
Be the first to comment!