Back to AI News
arxiv

Triplet Incremental Training Adapts General VLMs for Autonomous Driving

Researchers introduce a Triplet Incremental Training framework that instills 3D geometric awareness into general vision-language model backbones for autonomous driving without architectural overhauls. The method uses automatically constructed image-depth-QA triplets in a tri-stream pipeline: an unfrozen primary VLM for driving tasks, an auxiliary depth stream for implicit geometry learning, and a frozen twin VLM with logit-level KL distillation to prevent semantic drift. The depth branch is discarded at deployment, so the added geometric awareness incurs no inference overhead. Across four benchmarks, the approach reportedly outperforms state-of-the-art driving VLMs, rivals domain-specific experts in 3D perception, and retains general multimodal reasoning.

AI Brief

It offers a non-intrusive, deployment-ready method for turning general VLMs into capable driving models while preserving open-world reasoning and avoiding inference overhead.

arXiv:2611.16727v1 Announce Type: cross Abstract: Adapting general Vision-Language Models (VLMs) to Autonomous Driving (AD) is thought to require heavy 3D spatial priors or intrusive architectural overhauls to compensate for a lack of spatial grounding. In this work, we present a counterintuitive pilot study: off-the-shelf general VLMs already exhibit remarkable adaptability across diverse driving tasks, while precise metric depth remains the primary missing link, a bottleneck readily addressable via onboard sensors and depth foundation models. To this end, we introduce the Triplet Incremental Training Framework, a non-intrusive paradigm that seamlessly instills 3D geometric awareness into a general VLM backbone while fully preserving its open-world capabilities. Driven by automatically constructed (image, depth, QA pair) triplets, our framework operates via a unified tri-stream training pipeline: an unfrozen primary VLM stream to adapt to driving tasks, an auxiliary depth stream that guides the visual backbone to implicitly imagine 3D scene geometry via explicit geometric constraints, and a frozen twin VLM stream employing logit-level KL distillation to prevent semantic drift. Crucially, the auxiliary depth branch is discarded during deployment, ensuring the enhanced geometric awareness comes with zero inference overhead. We evaluate our framework across four major benchmarks targeting four diverse capabilities spanning 3D spatial perception, driving-oriented language reasoning, end-to-end planning, and general multimodal understanding. Our approach outperforms state-of-the-art driving VLMs, rivals domain-specific expert models in 3D perception, and successfully retains its original general reasoning capabilities. This demonstrates that Triplet Incremental Training offers a unified, deployment-ready paradigm for transforming general foundation models into robust driving intelligences.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!