Back to AI News
arxiv

O²LaF Repurposes Video Diffusion Transformers for Latent Action Learning

Researchers introduce O²LaF, a method that initializes both inverse and forward dynamics models from the same pretrained video diffusion transformer. One model acts as a feed-forward transition encoder with learnable action queries, while the other is a causal, action-conditioned video generator. The framework is trained end-to-end through a compact bottleneck to learn transferable actions from unlabeled videos. The authors report state-of-the-art few-shot action probing and stronger zero-shot action transfer across visual contexts and embodiments, with competitive results also shown using V-JEPA2 priors.

AI Brief

It offers a concrete research approach for extracting transferable action representations from pretrained video models without labeled action data.

arXiv:2611.26617v1 Announce Type: cross Abstract: Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O$^{2}$LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O$^{2}$LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O$^{2}$LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!