arXiv:2611.26366v1 Announce Type: new
Abstract: World models are increasingly used to simulate physical interactions, but training a separate model for each environment limits scalability. Cross-environment prediction requires identifying the current physical rule, which appearance alone may not reveal. We therefore study visual calibration: using object motion and interactions in a short video as direct evidence to infer the current rule and predict motion under that rule from a different initial state. However, differences in data availability across physical conditions can yield a long-tailed training distribution, biasing models toward common rules even when evidence supports rare ones. To study this challenge, we introduce TailWorld, which evaluates the transfer of demonstrated rules across initial states while independently controlling training imbalance and calibration informativeness. The benchmark compares predictions with two reference futures from the same prediction initial state, one following the demonstrated rule and the other a common rule, to measure prediction bias toward common dynamics. We further propose Rule Inference for Transfer (RIFT), which learns how different rules affect object motion, infers the current rule from calibration interactions, and applies it to prediction from a new initial state. RIFT explicitly controls the influence of training frequency on rule inference, allowing sufficiently informative observational evidence to support rare rules. Experiments on TailWorld show that RIFT improves prediction accuracy and physical consistency across varying training imbalances and distribution shifts, with gains in predicting rare dynamics.
arxiv
TailWorld and RIFT target rule inference in world models
The paper introduces TailWorld, a benchmark for cross-environment world-model prediction that independently controls training imbalance and the informativeness of visual calibration. It also proposes Rule Inference for Transfer (RIFT), which learns how different physical rules affect object motion, infers the current rule from short calibration videos, and applies it to prediction from a new initial state. Experiments on TailWorld reportedly show improved prediction accuracy and physical consistency under varying training imbalances and distribution shifts, including gains for rare dynamics.
It offers a concrete benchmark and method for making world models infer physical rules from visual evidence instead of defaulting to common dynamics.
Comments & Questions (0)
No comments yet
Be the first to comment!