Back to AI News
arxiv

Over-Realization in Text-to-Image Models: New Benchmark Shows Alignment Metrics Reward the Failure

This paper identifies 'over-realization' in text-to-image generation, where a prompt-mentioned target appears in ways the prompt did not request, such as under negation, in imagined or believed content, in figurative comparisons, or behind occluding viewpoints. The authors introduce OverReal, comprising OverReal-Gen with 698 prompts across four situations and OverReal-Det with 2,841 human-labeled generated images. Evaluating eight text-to-image systems, they find all over-realize and that conventional metrics like CLIPScore reward rather than penalize the failure. Prompt expansion only partially mitigates the issue, and attention analysis shows cue words meant to suppress targets receive less attention than ordinary words.

AI Brief

Provides a concrete new benchmark and systematic evaluation exposing a blind spot in text-to-image alignment metrics, useful for researchers and model developers.

arXiv:2611.16630v1 Announce Type: cross Abstract: In text-to-image generation, a target that the prompt mentions often appears in the generated image in a way the prompt does not call for. We call this failure over-realization. Mental-spaces theory points to the situations where it can happen: under a negation, inside what a character believes or imagines, as the vehicle of a figurative comparison, or behind a viewpoint that hides the target. Prior work has studied only negation in depth, and the conventional alignment and preference metrics do not penalize the failure. We build OverReal to measure it: OverReal-Gen, 698 prompts covering the four situations, on which we measure each generator's over-realization rate; and OverReal-Det, 2,841 generated images human-labeled for whether and how the target is over-realized, on which we verify the automatic judge that produces this rate. Our evaluation of eight text-to-image systems reveals that all of them over-realize, and that conventional metrics such as CLIPScore reward the failure rather than penalize it. Prompt expansion only partly mitigates the failure. Inside the generator, the diffusion backbone attends to a mentioned target almost as much as to a requested one. The cue words that should hold the target back receive less attention than ordinary words.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!