WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

Citations Over Time

Abstract

Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction between image-text pairs, by assuming that there exists strong semantic correlation between the text and image modalities. Since this strong assumption is often invalid in real-world scenarios, we choose to implicitly model the cross-modal correlation for large-scale multi-modal pre-training, which is the focus of the Chinese project `WenLan' led by our team. Specifically, with the weak correlation assumption over image-text pairs, we propose a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. Unlike OpenAI CLIP that adopts a simple contrastive learning method, we devise a more advanced algorithm by adapting the latest method MoCo into the cross-modal scenario. By building a large queue-based dictionary, our BriVL can incorporate more negative samples in limited GPU resources. We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model. Extensive experiments demonstrate that the pre-trained BriVL model outperforms both UNITER and OpenAI CLIP on various downstream tasks.

Related Papers

• → Some considerations of bridging stresses for fiber-reinforced ceramics(1991)24 cited
• → Bridging the gap between science and politics—The role of precaution in chemicals control(2006)1 cited
• → The Art of Bridging(2023)1 cited
• → Testing for Bridging Faults(2009)3 cited
• → Bridging the Divide: Part 2 – Science Bridging Units(2008)