Back to AI News
arxiv

dNaViT proposes discrete visual tokens as a general-purpose representation

Researchers introduce dNaViT, a Discrete Native-resolution Vision Transformer that produces discrete visual primitives analogous to language tokens, built on a frozen pretrained encoder. The method uses lightweight residual modulation, a dual-branch objective, and grouped product quantization to preserve semantic and pixel-level information without inflating sequence lengths. The paper reports that dNaViT matches or exceeds task-specific baselines across 12 visual understanding benchmarks, 4 generation benchmarks, and an image editing benchmark. It is positioned as a versatile discrete visual tokenizer supporting unified multimodal encoding and high-fidelity image decoding.

AI Brief

It offers a concrete new visual tokenization method with broad benchmark coverage and implications for unified multimodal models.

arXiv:2611.12426v1 Announce Type: cross Abstract: Discrete autoregressive modeling, notably through next-token prediction, has become a cornerstone of modern language models. Extending this paradigm to vision allows visual and language models to converge on a unified framework, thereby fostering shared methodology and infrastructure. This work investigates a central question: how far can a finite discrete vocabulary go in representing the richness of the visual world? We present dNaViT, a Discrete Native-resolution Vision Transformer that produces discrete visual primitives analogous to language tokens. Built upon a frozen pretrained encoder, dNaViT applies lightweight residual modulation with a dual-branch objective to inject reconstruction-aware cues into semantic features. By leveraging grouped product quantization, it scales discrete capacity without inflating spatial sequence lengths, while a balanced training recipe maintains both semantic and pixel-level information. Consequently, dNaViT functions as a versatile discrete visual tokenizer, enabling unified encoding across downstream tasks alongside high-fidelity image decoding. Extensive evaluations show that dNaViT matches or exceeds task-specific baselines across 12 visual understanding benchmarks, 4 generation benchmarks, and an image editing benchmark. These findings demonstrate that a compact discrete vocabulary can serve as a universal visual currency and unlock the full potential of fully unified, native multimodal methodology.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!