Multi-Modality Cross Attention Network for Image and Sentence Matching

Citations Over TimeTop 1% of 2020 papers

Abstract

The key of image and sentence matching is to accurately measure the visual-semantic similarity between an image and a sentence. However, most existing methods make use of only the intra-modality relationship within each modality or the inter-modality relationship between image regions and sentence words for the cross-modal matching task. Different from them, in this work, we propose a novel MultiModality Cross Attention (MMCA) Network for image and sentence matching by jointly modeling the intra-modality and inter-modality relationships of image regions and sentence words in a unified deep model. In the proposed MMCA, we design a novel cross-attention mechanism, which is able to exploit not only the intra-modality relationship within each modality, but also the inter-modality relationship between image regions and sentence words to complement and enhance each other for image and sentence matching. Extensive experimental results on two standard benchmarks including Flickr30K and MS-COCO demonstrate that the proposed model performs favorably against state-of-the-art image and sentence matching methods.

Related Papers

• → Factors Influencing Modality Choice in Multimodal Applications(2008)15 cited
• → About the Teaching Strategies of Modality in the Classroom(2016)1 cited
• Brief comment on modality in English(2007)
• The Pragmatic Analysis of the Subject of "Bei"——sentence in Dunuang Bianwen(2010)
• A Preliminary Study on Modality of Chinese(2002)