VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Citations Over Time

Abstract

We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce Mixture-of-Modality-Experts (MoME) Transformer, where each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of MoME, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the-art results on various vision-language tasks, including VQA, NLVR2 and image-text retrieval. The code and pretrained models are available at https://aka.ms/vlmo.

Related Papers

• Overview of Question-Answering(2002)
• → Twitch of the Death Nerve (aka Reazione a catena [ Chain Reaction ], aka A Bay of Blood , aka Antefatto , Before the Fact – Ecology of a Crime , aka Bloodbath Bay of Death , aka Carnage , aka Last House on the Left Part II , aka New House on the Left , aka The Ecology of a Crime )(2007)
• → Kill, Baby . . . Kill! (aka Operazione paura [ Operation Fear ], aka Don’t Walk in the Park , aka Curse of the Dead )(2007)
• Chandra X-ray observations of SWIFT J0045.2+4151 (aka Sw J0045 aka GRB 150301C aka p[PFH2005] 622)(2015)