Back to AI News
arxiv

Multi-View Transformer with Geometry-Guided Multi-Modal ControlNet for 3D Virtual Try-On

Researchers introduce GS-VTON-MVT, a three-stage framework for consistent 3D Gaussian Splatting virtual try-on. It combines a multi-view inpainting transformer with full cross-view self-attention, a geometry-guided multi-modal ControlNet conditioned on garment images plus depth and surface normals, and pose-adaptive persona-aware attention. On the 3D-VTONBench benchmark, the method improves FID, KID, LPIPS-based cross-view consistency, and user preference over general-purpose 3D editors and prior 3D try-on methods.

AI Brief

It presents a concrete research method for reducing cross-view texture drift and geometric inconsistency in 3D virtual try-on.

arXiv:2611.25979v1 Announce Type: cross Abstract: Editing a 3D Gaussian Splatting (3DGS) capture of a person so that they wear a new garment requires every rendered view to agree on the same garment details. Existing 3D virtual try-on (VTON) pipelines, however, edit training views largely independently and fuse them with hand-designed heuristics such as manually chosen reference views and fixed attention-blending weights. The resulting inconsistencies are baked into the 3D representation and surface as texture drift, blurred prints, and implausible geometry in novel views. We present GS-VTON-MVT, a three-stage framework that combines implicit and explicit 3D reasoning for consistent 3DGS try-on. (i) A multi-view inpainting transformer jointly edits a set of training views with full cross-view self-attention, so that cross-view correspondence is learned rather than imposed. (ii) A geometry-guided multi-modal ControlNet conditions the diffusion editor on the garment image together with depth and surface normals rendered from the current 3DGS state, supplying explicit structure where visual evidence is weakest. (iii) A pose-adaptive persona-aware attention replaces the fixed blending weight between current-view and reference-view features with weights derived from camera-pose proximity. On the 3D-VTONBench benchmark, GS-VTON-MVT improves FID, KID, LPIPS-based cross-view consistency, and user preference over general-purpose 3D editors and prior 3D try-on methods.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!