Back to AI News
arxiv

ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images

ProtoDiffusion is a latent-diffusion framework built on ProtoVAE, a multi-layer variational autoencoder that encodes a design into one shared spatial prototype plus one candidate vector per layer. This yields HW+N tokens for N layers before backbone packing, compared with NHW for full-grid-per-layer representations. The model jointly denoises the prototype and a fixed set of candidate slots conditioned on an image or text, with bounding boxes as optional layout guidance, and supports both image-to-design and text-to-design. Reported results include 40.43 dB PSNR on layer crops, image-to-design RGB L1 scores of 0.040 with boxes and 0.057 without, and text-to-design FID of 66.41 versus 86.84 for FLUX.2 [klein], while running 19.1x faster than a generation-then-decomposition cascade.

AI Brief

It introduces a more compact latent representation for layered graphic-design generation and reports concrete reconstruction, layout and text-to-design benchmark results.

arXiv:2611.26586v1 Announce Type: new Abstract: Graphic designs are ordered compositions of RGBA layers whose separation enables selection, movement, and reuse. Existing layered models represent each layer with a full-canvas or region-sized spatial latent grid, so representation size grows with the cumulative spatial extent of the layers. We present ProtoDiffusion, a latent-diffusion framework built on ProtoVAE, a multi-layer variational autoencoder that encodes a design into one shared spatial prototype and one candidate vector per layer. Each candidate conditions a shared decoder to reconstruct its corresponding RGBA layer. For $N$ layers, the representation contains $HW+N$ tokens before backbone-specific packing, compared with $NHW$ for full-grid-per-layer representations. ProtoDiffusion jointly denoises the prototype and a fixed set of candidate slots, conditioned on an image or text, with bounding boxes as optional layout guidance. The model leaves unneeded slots empty and determines the number of layers and, in absence of boxes, their layout, requiring neither as input. The same architecture supports both image-to-design and text-to-design. ProtoVAE achieves high-fidelity reconstruction from its compact shared latent representation, reaching 40.43 dB PSNR on layer crops. On the LayerD evaluation split, image-to-design achieves RGB L1 scores of 0.040 with bounding boxes and 0.057 without, with box-guided alpha soft IoU (0.850) comparable to the strongest fine-tuned baseline (0.871). On merged images, text-to-design reaches 66.41 FID versus 86.84 for FLUX.2 [klein], achieves a 0.860 text-accuracy score, and runs $19.1\times$ faster than the same-backbone generation-then-decomposition cascade.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!