V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
The paper argues that world-model video encoders such as V-JEPA encode scene dynamics (direction, speed, order, state) in a subspace shared across videos, while frame-wise text-aligned encoders like SigLIP do not, and that caption/Q&A-trained language models tend to retrieve named attributes instead of reading the clip, leaving dynamics unused. It introduces counterfactual supervision, using pixel transformations to turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. The resulting V-JEPA-LM models (1B to 8B, built on Qwen3 and Llama) were trained in a single stage on 4.53M public samples with 32 visual tokens per frame, and are reported to lead every size class in average score across eleven video benchmarks. The 4B model scores 72.1 versus 61.1 for the best open 4B and 64.8 for the best open 8B, and a controlled encoder x supervision study reports counterfactual supervision adding 4.5 points on average and 38 on speed questions.
Why it matters: It offers a concrete method and open models for making video-language systems actually use encoded motion dynamics, with head-to-head benchmark numbers against comparable open models.