arXiv:2611.06627v1 Announce Type: new
Abstract: We present a shared framework that adapts Cosmos world foundation models for real-time interactive world simulation in Physical AI. By streaming visual observations in response to actions and scene controls, the framework enables users and embodied agents to interact with evolving virtual environments. It supports both predicting the visual consequences of actions and generating observations from physical state maintained by an external simulator. We demonstrate this approach across camera navigation and robot manipulation, controllable video transfer, and multi-view autonomous vehicle (AV) simulation. These examples provide a shared foundation for developing interactive virtual environments for Physical AI training and evaluation. Our code and model checkpoints are available at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3-examples. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation.
arxiv
Interactive World Simulation with Cosmos for Physical AI
NVIDIA introduces a framework adapting Cosmos world foundation models for real-time interactive simulation in Physical AI. It streams visual observations in response to actions and scene controls, supporting both action-conditioned prediction and external simulator state. Demonstrations include camera navigation, robot manipulation, controllable video transfer, and multi-view AV simulation. Code and checkpoints are publicly available.
Provides a concrete, recent framework for interactive world simulation, directly useful for AI training and evaluation in embodied and autonomous systems.
Comments & Questions (0)
No comments yet
Be the first to comment!