Back to AI News
arxiv

The balance between feature learning and collapse in dynamical systems-based generative models

This theoretical paper uses dynamical systems and operator theory to analyze gradient flow for a general class of stochastic differential equations used in continuous-time generative models. It links mode collapse to low-rank biases and identifies two mechanisms: collapsed models as stable fixed points, and sequential feature learning in which smaller modes take progressively longer to learn. The authors frame these findings as a trade-off between feature learning and collapse and propose an operator view toward a unified learning theory for models such as normalizing flows and diffusion.

AI Brief

It offers a theoretical explanation of mode collapse in continuous-time generative models, giving researchers a framework for understanding training dynamics and representation learning.

arXiv:2611.12156v1 Announce Type: cross Abstract: Over the past decade, a growing body of work has established that implicit biases in learning dynamics fundamentally shape the solutions found by neural networks, governing their learned features and generalisation performance. However, despite these strides in deep learning theory, far less is known about representation learning in generative models based on dynamical systems, from normalising flows to diffusion. Importantly, empirical evidence suggests that continuous-time generative models can be prone to mode collapse, yet a learning theory explaining such biases remains elusive. To address this, we use tools from dynamical systems and operator theory to express the gradient flow of a general class of stochastic differential equations as a composition of linear operators, allowing us to relate collapse to low-rank biases in generative models. In variational inference and flow-matching tasks, we show that this bias can manifest in two different mechanisms: i) where collapsed models are stable fixed points of the gradient flow, or ii) in sequential feature learning, where smaller modes take progressively longer to be learned, producing an effective collapse for a fixed training budget. Together these findings point to a trade-off between feature learning and collapse, suggesting an operator view as a step towards a unified framework for understanding the learning dynamics of dynamical systems-based generative models.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!