Back to AI News
arxiv

Alignment by Confident Debate: Preference Learning from Multi-Agent Debate

AbCD converts multi-agent debate traces into preference data without gold-answer labels. It clusters final answers by task-level semantic equivalence and combines answer equivalence with confidence-weighted disagreement. Tested on mathematical reasoning, multiple-choice quantitative QA, and executable code generation, post-training on debate-derived preferences outperformed inference-only ensembling, debate, and SFT on the same traces across five model families from 1B to 8B parameters and four datasets. The work reports a maximum consensus-accuracy gain of 21.5 percentage points and also examines limitations of heterogeneous committees and agreement-based weighting.

AI Brief

It offers a concrete self-supervised preference-learning method for turning multi-agent debate into training supervision, with controlled evaluations across reasoning and code tasks.

arXiv:2611.09568v1 Announce Type: new Abstract: Post-training language models for reasoning typically rely on gold answers, task-specific verifiers, or learned reward models, limiting scalability when such signals are scarce. Multi-agent debate offers a self-supervised alternative, but converting debate into reliable training supervision remains challenging: majority voting weights agreeing responses equally, lacks a principled tie-breaking rule, and is ill-defined for open-ended tasks without a canonical answer; likelihood-based scoring can conflate fluency with correctness; and homogeneous committees may reinforce correlated errors. We introduce Alignment by Confident Debate (AbCD), which converts debate traces into preference data without gold answer labels. A sampling-diverse committee debates each problem, after which final answers are clustered by task-level semantic equivalence. We use verifiable tasks (mathematical reasoning, multiple-choice quantitative QA, and executable code-generation) as a controlled test bed: their answer keys or executable checkers let us evaluate the quality of debate-derived supervision against ground truth, even though the oracle itself never uses gold labels. Post-training on the resulting preferences outperforms inference-only ensembling and debate, as well as SFT on the same traces, across five model families spanning 1B to 8B parameters and four datasets, improving consensus accuracy, with a maximum gain of $21.5_{pp}$. Ablations show that the approach transfers to a larger model and to code generation, while also revealing limitations of heterogeneous committees and agreement-based weighting. Overall, the results suggest that debate traces can yield useful preference supervision when answer equivalence is combined with confidence-weighted disagreement.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!