Research
From Points to Trajectories: Rethinking Multimodal Alignment for Physical Events
This paper argues that compressing temporal sequences into pooled embeddings erases transient cross-modal dynamics, causing distinct physical executions under identical task conditions to be confused. It formally shows that temporally pooled representations converge to the stationary recording condition as sequence length grows, limiting what any loss or embedding geometry can disambiguate. The authors introduce CounterPoint, a lightweight plug-in that removes each modality's temporal mean and scores frame-level cross-modal agreement. Integrated into four state-of-the-art multimodal aligners, it is reported to consistently outperform existing temporal alignment methods.
multimodal alignmentphysical AItemporal modeling