arXiv:2611.16767v1 Announce Type: cross
Abstract: Natural language explanations promised by modern Video Large Language Models (Video LLMs) offer autonomous driving systems a path toward detailed scene description and interpretable decision making on the road. While recent benchmarks have begun assessing these models on driving scenes, they mainly target routine traffic conditions and general comprehension tasks based on existing driving datasets and therefore fail to evaluate model performance on the rarest and most safety-critical driving events, also known as "corner cases". To provide a comprehensive evaluation of Video LLMs on precisely these scenarios, we introduce DashBench, a benchmark testing 25 Video LLMs on 1,709 carefully curated driving videos spanning not only collision-related corner cases but also rare non-collision ones. Through extensive prompts and open-ended question answering, we expose a critical capability gap: all evaluated models perform substantially below the threshold required for safety-critical driving applications, frequently misjudging or missing dangerous events on the road ahead and hallucinating non-existent agents or actions in their explanations. We believe DashBench, combining an extensive dataset with a rigorous evaluation toolkit and a detailed diagnosis of current model limitations, drives progress toward safer, more explainable autonomous driving systems. The code and data will be shared at: cornercase-dashbench.github.io
arxiv
Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving
Researchers introduce DashBench, a benchmark for evaluating Video Large Language Models on rare and safety-critical driving scenarios. It tests 25 Video LLMs on 1,709 curated driving videos covering collision-related and rare non-collision corner cases through open-ended question answering. The evaluation finds that all tested models perform substantially below the threshold required for safety-critical driving applications, often missing or misjudging dangerous events and hallucinating agents or actions. The work aims to support progress toward safer and more explainable autonomous driving systems.
It provides a concrete benchmark and evidence that current Video LLMs remain unreliable for safety-critical autonomous driving corner cases.
Comments & Questions (0)
No comments yet
Be the first to comment!