Back to AI News
arxiv

Closed-Loop World Model Couples Driving Behavior and Visual Rendering

Researchers present a closed-loop interactive world model for autonomous driving that couples multi-agent behavior modeling with image-based rendering. Background-vehicle policies are fine-tuned with multi-agent reinforcement learning against an end-to-end ego planner that reacts to rendered images. A per-Gaussian diagnosis procedure attenuates view-dependent colors on under-observed Gaussians and fills never-observed surfaces, while signal lights are rendered from learned vehicle intentions. The paper reports an 85% reduction in divergence from real-world interaction behaviors and a 2.0 dB novel-view vehicle PSNR improvement over a state-of-the-art world model on long-tail closed-loop scenarios.

AI Brief

It offers a concrete method for making simulated driving interactions behaviorally and visually realistic, which matters for training and evaluating autonomous systems in unseen scenarios.

arXiv:2611.22691v1 Announce Type: cross Abstract: World models (WMs) generate driving scenarios beyond recorded logs, enabling the training and evaluation of autonomous driving systems in unseen interactions. However, existing WMs struggle to ensure both behavioral realism and visual fidelity, since they overlook the relationship between vehicle interactions and the ego visual observations. We propose a closed-loop interactive WM that couples behavior modeling and visual rendering. First, imitation-learned background-vehicle (BV) policies are fine-tuned with multi-agent reinforcement learning in the closed loop against an end-to-end ego planner that reacts to the rendered images, so that their interaction behaviors match those in real logs. Second, the viewpoints exposed by these interactions guide a per-Gaussian diagnosis of each 3D Gaussian vehicle asset, which attenuates view-dependent colors on under-observed Gaussians and fills never-observed surfaces with generated Gaussians, while preserving the observed appearance. Third, signal lights are rendered according to the driving intentions of the vehicles, by an editor that learns the light states unobserved for a vehicle through counterfactual fine-tuning with a light-state classifier. On long-tail closed-loop scenarios, our method reduces the divergence from real-world interaction behaviors by 85% and improves novel-view vehicle PSNR by 2.0dB over the state-of-the-art WM. The significance of BV light signaling in WMs is shown, as the rendered signal lights conveying BV driving intentions indirectly impact ego driving behaviors in a quarter of the scenarios.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!