Research Editor selected
RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
RoboTryOn converts egocentric human videos into robot demonstration videos by swapping the human for a target robot while preserving the task, objects, and background. It uses autoregressive keyframe translation and motion conditioning on human hand skeletons, plus an inverse dynamics model with depth features to infer actions. Policies trained on this synthetic data improved success rates on novel bimanual tasks from 2% to 60% without real robot demos, and matched teleoperation-trained performance at about a third of the collection cost.
Why it matters: This paper presents a novel, cost-effective method for generating robot training data from human videos, with strong experimental results, making it valuable for robotics and AI research.
Robot foundation modelsImitation learningData augmentation