Back to AI News
arxiv

PACT trains GUI agents to align executed actions with correct plans

PACT is a training framework for GUI agents that targets cases where an agent forms a correct plan but emits an action that fails to execute it. It protects reference-aligned plan tokens from negative policy-gradient updates and uses a preference surrogate to favor reference actions over erroneous emitted actions without changing the original scalar action reward. On AndroidWorld, PACT improves task success from 60.34% to 68.97%, and on AndroidControl High it improves rule-based plan-action consistency from 83.71% to 89.74%. Adapted to desktop agents, it raises task success on the 369-task OSWorld benchmark from 40.92% to 44.17%.

AI Brief

It offers a concrete credit-assignment method for reducing GUI-agent execution errors, supported by benchmark improvements across mobile and desktop environments.

arXiv:2611.22591v1 Announce Type: new Abstract: GUI agents can formulate a correct immediate plan in their reasoning yet emit an action that fails to execute it, creating an execution gap. This recurring mismatch undermines reliable task completion, as a single execution error can derail a multi-step workflow. In outcome-supervised reinforcement learning, broadcasting a negative response-level advantage across all generated tokens risks penalizing valid planning together with erroneous execution. To address this credit-assignment conflict, we introduce PACT (Plan-Action Consistency Training), a training framework that combines reference-aligned plan protection and plan-aligned action repair without altering the original scalar action reward. Using annotated reference actions to identify execution gaps, PACT protects the verified plan by masking negative policy-gradient contributions across its token span. Keeping the generated reasoning unchanged as context, it then favors the reference action over the emitted erroneous action through a preference surrogate with stop-gradient pair weights. Compared with the outcome-only baseline, PACT raises AndroidWorld task success from 60.34% to 68.97% (+8.63 pp), while improving rule-based plan-action consistency on AndroidControl High from 83.71% to 89.74% (+6.03 pp). Adapting PACT to desktop agents also improves task success from 40.92% to 44.17% (+3.25 pp) on the full 369-task OSWorld benchmark. Together, these results establish PACT as an effective credit-assignment framework for bridging execution gaps and improving task completion across mobile and desktop GUI environments.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!