Back to AI News
arxiv

ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain

ZeroVLN is a slow-fast vision-language navigation framework that avoids navigation-domain supervised fine-tuning of the VLM brain. Instead of predicting geometric goals, the VLM uses a semantic tool-calling interface to send movement decisions in natural language to an action expert. The action expert is first adapted to execute these requests, after which closed-loop reinforcement learning jointly improves movement decisions and continuous execution. The authors report that ZeroVLN outperforms representative SFT-based hierarchical VLN methods on VLN-CE benchmarks while only minimally changing the VLM brain and largely preserving its general capabilities.

AI Brief

It offers a concrete research direction for building capable navigation agents without degrading the base VLM through domain-specific fine-tuning.

arXiv:2611.26414v1 Announce Type: new Abstract: Representative vision-language navigation (VLN) methods typically couple a vision-language model (VLM) brain with a learnable action head and fine-tune the VLM brain on navigation-domain data to predict geometric goals. However, such adaptation can substantially alter base VLM weights and degrade general visual capabilities. We introduce ZeroVLN, a slow-fast navigation framework that achieves effective navigation with zero navigation-domain SFT of the VLM brain. Rather than teaching the brain to predict geometric goals, we equip it with a semantic tool-calling interface for conveying movement decisions to an action expert in natural language. We first adapt the action expert to carry out these movement requests, then jointly improve the brain's movement decisions and the action expert's continuous execution through closed-loop reinforcement learning (RL). Trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks, while inducing only minor changes to the brain's weights and largely preserving its general capabilities. To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!