SemPair Relaxes Cross-Modal Correspondence for Remote Sensing Pretraining
The paper challenges the assumption that multimodal remote sensing pretraining requires exact instance-level cross-modal correspondence. It introduces Semantic Pairing (SemPair), which substitutes class-consistent cross-instance pairs for exact instance pairs, and a latent-factor model separating shared semantic structure from instance-specific correlation. A controlled interpolation, Mixed Pairing (MixPair), is used to study correspondence granularity, and experiments show SemPair reaches transfer performance comparable to exact pairing while MixPair reveals a non-monotonic relationship between downstream transfer and instance-pairing ratio. The authors conclude that exact instance correspondence is not a prerequisite for effective multimodal pretraining.
It offers a concrete pretraining strategy that could reduce reliance on costly instance-aligned multimodal data, relevant to researchers working on multimodal representation learning.
Comments & Questions (0)
No comments yet
Be the first to comment!