Back to AI News
arxiv

ModHarness-RL treats agentic RL harnesses as learnable, evolving programs

ModHarness-RL reframes the harness around an agentic model — the scaffold controlling what the model sees of its interaction and what guidance it gets — as programs learned from data rather than hand-written and fixed. Interactions are recorded as structured State made of typed blocks, with a Compression program assembling history into context and a Steering program supplying guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories so the harness tracks shifts in model behavior. Experiments on CodeSearch, plus ALFWorld and BIRD SQL with models from 1.5B to 8B, show the Compression and Steering mechanisms improve held-out performance over a fixed-harness baseline in most settings.

AI Brief

It offers a concrete method for co-evolving the agent scaffold with the policy during RL, which is useful for researchers and engineers building agentic training pipelines.

arXiv:2611.18533v1 Announce Type: new Abstract: Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!