Back to AI News
arxiv

JointGV: Graph-Based Correction for Long-Horizon Visual World Models

JointGV is a plug-in method that rolls out a task-conditioned scene graph alongside a visual world model and couples their predicted changes to compute a bounded correction for accumulated pose and spatial-relation drift. The authors attach it to DINO-WM and DreamerV3 and evaluate on DMC Cheetah Run, Walker Walk, and contact-rich Finger Turn Hard across three seeds. Over 32-step rollouts, they report 11–28% lower joint-position error. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from 12.56° to 6.69°, and an actor trained with corrected rollouts completes 46 of 50 simulated control episodes versus none with original rollouts.

AI Brief

It offers robotics and world-model researchers a plug-in correction that reduces long-horizon visual prediction drift and improves downstream control without restructuring the original visual model.

arXiv:2611.22543v1 Announce Type: cross Abstract: Visual world models predict future visual states conditioned on actions, but recursive errors in entity poses and spatial relations limit their long-horizon prediction for robot planning and control. We ask whether a separately predicted relational state can correct this drift without restructuring the visual model. This is challenging because graph nodes and visual states share neither elements nor a feature space. We propose Joint Graph--Visual Dynamics Correction (JointGV), a plug-in based on a cross-granularity dynamics hypothesis: despite their different representations, graph-node and visual-state changes describe the same physical transition. JointGV rolls out a task-conditioned scene graph alongside the visual world model, couples their predicted changes, and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded residual. Graph-learning and correction gradients do not propagate into the original visual dynamics. We attach JointGV to DINO-WM (spatial tokens) and DreamerV3 (a global latent state), evaluating both on DMC Cheetah Run and Walker Walk across three seeds and extending the evaluation to contact-rich manipulation with DMC Finger Turn Hard. Across 32-step rollouts, JointGV improves structural and visual-prediction accuracy, reducing joint-position error by 11--28\%. For spatial tokens, learned graph--visual change coupling gives lower error than broadcasting the same mean graph-motion context to every token, particularly when non-adjacent parts appear close together in the image. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from $12.56^\circ$ to $6.69^\circ$ and improves DreamerV3 and LeWorldModel as well. In simulated inverted-pendulum control, an actor trained with JointGV-corrected rollouts completes 46 of 50 episodes, compared with none using the original rollouts.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!