Over-Realization in Text-to-Image Models: New Benchmark Shows Alignment Metrics Reward the Failure
This paper identifies 'over-realization' in text-to-image generation, where a prompt-mentioned target appears in ways the prompt did not request, such as under negation, in imagined or believed content, in figurative comparisons, or behind occluding viewpoints. The authors introduce OverReal, comprising OverReal-Gen with 698 prompts across four situations and OverReal-Det with 2,841 human-labeled generated images. Evaluating eight text-to-image systems, they find all over-realize and that conventional metrics like CLIPScore reward rather than penalize the failure. Prompt expansion only partially mitigates the issue, and attention analysis shows cue words meant to suppress targets receive less attention than ordinary words.
Provides a concrete new benchmark and systematic evaluation exposing a blind spot in text-to-image alignment metrics, useful for researchers and model developers.
Comments & Questions (0)
No comments yet
Be the first to comment!