Back to AI News
arxiv

HiUni: A Unified Multimodal Model Sharing One Backbone for Understanding and Generation

HiUni is a unified multimodal model that reuses the same language-aligned ViT and dense Transformer backbone for both visual understanding and image generation, confining task-specific computation to the output side. To handle pixel-space optimization and the loss of fine spatial detail in pretrained ViT features, it introduces a Pixel Diffusion Head and a lightweight HiGate pathway. Experiments report an average understanding score similar to the pretrained VLM while achieving competitive image generation and editing performance. The authors argue a single dense backbone can support both capabilities without task-specific backbone computation.

AI Brief

It offers a concrete architectural result on simplifying unified multimodal models by sharing one backbone across understanding and generation.

arXiv:2611.26107v1 Announce Type: cross Abstract: Unified multimodal models aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of these two tasks, which limits the degree of unification. In this paper, we present HiUni, which revisits this design choice and explores whether a pretrained vision language model can be directly extended into a unified model with extensive parameter sharing. Specifically, HiUni reuses the same language-aligned ViT and dense Transformer backbone for both capabilities, while confining task-specific computation to the output side. Yet such a design introduces two key challenges. Generation in pixel space is difficult to optimize, and semantic abstraction in pretrained ViT discards fine spatial details required for synthesis. To address these challenges, HiUni introduces a \emph{Pixel Diffusion Head} tailored to direct pixel-space optimization and a lightweight \emph{HiGate} pathway that provides fine-grained spatial details near the output stage. Extensive experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons and analyses further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!