Back to AI News
arxiv

ReFrain Adds Abstention to Multimodal Models Without Losing Correct Answers

Researchers propose ReFrain (Regression-Free Abstention), which combines a linear probe on first-token logits, threshold calibration on held-out image clusters, and a deterministic veto that replaces rejected outputs with a displayed "none of the above" option. On Qwen2.5-VL-3B at a zero-budget target, ReFrain retained all 57 SFT-gained answers with no losses while improving correct abstention by 7.30 percentage points. At a 0.5% calibration budget it reached 74.05% correct abstention with 8 losses, versus CLIP-UP EmbLoRA at 64.65% with 62 losses. Confirmation on 228 independent question clusters yielded 66.05% correct abstention with no losses at the 0.5% point.

AI Brief

It offers a practical method for making multimodal models refuse unanswerable questions without sacrificing previously learned correct responses, backed by measured comparisons and released code.

arXiv:2611.14311v1 Announce Type: new Abstract: Adding abstention to a task-adapted multimodal model can discard correct answers, including capabilities acquired during adaptation. We formulate abstention repair as maximizing correct abstention on unanswerable questions under a budget on newly lost correct answers. We propose ReFrain (Regression-Free Abstention), which combines a linear probe on first-token logits, threshold calibration on held-out image clusters, and a deterministic veto that replaces rejected outputs with the displayed "none of the above" option while preserving all other responses. "Regression-free" denotes zero loss on the calibration set at the zero-budget target. All abstention methods in our main comparison start from the same answering-SFT checkpoint; we separately track lost correct answers, repaired errors, and retained SFT gains. On Qwen2.5-VL-3B, ReFrain at zero budget retains all 57 SFT-gained answers with no losses and improves correct abstention by 7.30 percentage points. At a 0.5% calibration budget, it reaches 74.05% correct abstention with 8 losses and retains 55 gains; CLIP-UP EmbLoRA reaches 64.65% with 62 losses, 41 repairs, and 40 retained gains. The same probe at its default threshold loses 298 correct answers. With models and thresholds frozen, confirmation on 228 independent question clusters yields 66.05% correct abstention and no losses at the 0.5% point, exceeding EmbLoRA by 32.72 percentage points. Ordinary refusal SFT achieves higher correct abstention at greater regression cost. Code and reproduction materials: https://github.com/ModeLiberaL/ReFrain.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!