Back to AI News
arxiv

Two-channel constrained GRPO calibrates VLM safety budgets

Researchers introduce two-channel constrained GRPO to enforce safety budgets on harmful assistance and over-refusal in vision-language models. The method addresses two failure modes: scorer mean bias in the Lagrange multiplier and within-group reward standardization that cancels the multiplier before it reaches the gradient. On a Qwen3-VL-2B policy, held-out harm on 1,037 adversarial prompts falls from 8.9% under standard GRPO to 4.8%, versus 7.6% for an uncalibrated budget, while over-refusal stays at most 2% on 400 benign queries and MMBench is unchanged. Gains are larger when intent is in the image and on the unseen HADES benchmark, with results holding across seeds, alternative judges and a human audit.

AI Brief

It offers a concrete constrained-RL fix for calibrating VLM safety behavior, with measured reductions in harmful assistance and limited over-refusal.

arXiv:2611.16628v1 Announce Type: new Abstract: Reinforcement learning can teach vision-language models (VLMs) to decline harmful image-text requests, but how safe the result becomes is set indirectly, through reward weights found by trial runs. Stating safety as budgets on the rate of harmful assistance and the rate of over-refusal, enforced by Lagrange multipliers inside GRPO with an off-the-shelf scorer as the cost, is the natural fix, and it fails twice. The multiplier compares the scorer's mean with the budget, and scorers built for ranking get the mean wrong: a fixed-point analysis shows that the mean bias, not the budget, then decides whether the multiplier settles, saturates or switches off. And standard GRPO standardizes the penalized reward within each group, which cancels the multiplier before it reaches the gradient. Two-channel constrained GRPO repairs both: the base reward keeps the raw score for the within-group comparison, the penalty and the constraint estimate pass through a monotone map fitted on judge-labeled training responses, and the advantage keeps the multiplier. On a Qwen3-VL-2B policy with a training-free readout of an open 4B model as the scorer, held-out harm on 1,037 adversarial prompts falls from 8.9% under standard GRPO to 4.8%, lower in every seed, where the uncalibrated budget gives 7.6%; over-refusal stays at most 2% on 400 benign queries and MMBench is unchanged. The gain is largest where the intent sits in the image (10.1% -> 5.1%) and grows on the unseen HADES benchmark (28.9% -> 12.8%). Order-preserving distortions of the scores reproduce each predicted controller state in every seed, and the results hold under a map fitted on another judge's labels, an image-reading judge and a human audit.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!