Back to AI News
arxiv

Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving

Researchers introduce DashBench, a benchmark for evaluating Video Large Language Models on rare and safety-critical driving scenarios. It tests 25 Video LLMs on 1,709 curated driving videos covering collision-related and rare non-collision corner cases through open-ended question answering. The evaluation finds that all tested models perform substantially below the threshold required for safety-critical driving applications, often missing or misjudging dangerous events and hallucinating agents or actions. The work aims to support progress toward safer and more explainable autonomous driving systems.

AI Brief

It provides a concrete benchmark and evidence that current Video LLMs remain unreliable for safety-critical autonomous driving corner cases.

arXiv:2611.16767v1 Announce Type: cross Abstract: Natural language explanations promised by modern Video Large Language Models (Video LLMs) offer autonomous driving systems a path toward detailed scene description and interpretable decision making on the road. While recent benchmarks have begun assessing these models on driving scenes, they mainly target routine traffic conditions and general comprehension tasks based on existing driving datasets and therefore fail to evaluate model performance on the rarest and most safety-critical driving events, also known as "corner cases". To provide a comprehensive evaluation of Video LLMs on precisely these scenarios, we introduce DashBench, a benchmark testing 25 Video LLMs on 1,709 carefully curated driving videos spanning not only collision-related corner cases but also rare non-collision ones. Through extensive prompts and open-ended question answering, we expose a critical capability gap: all evaluated models perform substantially below the threshold required for safety-critical driving applications, frequently misjudging or missing dangerous events on the road ahead and hallucinating non-existent agents or actions in their explanations. We believe DashBench, combining an extensive dataset with a rigorous evaluation toolkit and a detailed diagnosis of current model limitations, drives progress toward safer, more explainable autonomous driving systems. The code and data will be shared at: cornercase-dashbench.github.io
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!