Back to AI News
arxiv

Training LLMs to Follow Reasoning Instructions with Minimal Edits

Researchers propose edit-based training, a framework that learns from a model's own trajectories through localized textual corrections rather than full-trajectory imitation or trajectory-level reinforcement learning. An editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span, so supervision is concentrated only on edited spans; the method can be applied offline or online. On the ReasonIF benchmark with Qwen3-32B, the approach reportedly improved reasoning-instruction compliance while largely preserving performance on untargeted tasks compared with baselines.

AI Brief

It offers a targeted training alternative that may improve reasoning-instruction compliance with less collateral skill loss than full-trajectory SFT or trajectory-level RL.

arXiv:2611.26342v1 Announce Type: new Abstract: Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!