IMF: Interactive Multimodal Fusion Model for Link Prediction

Citations Over TimeTop 1% of 2023 papers

Abstract

Link prediction aims to identify potential missing triples in knowledge graphs. To get better results, some recent studies have introduced multimodal information to link prediction. However, these methods utilize multimodal information separately and neglect the complicated interaction between different modalities. In this paper, we aim at better modeling the inter-modality information and thus introduce a novel Interactive Multimodal Fusion (IMF) model to integrate knowledge from different modalities. To this end, we propose a two-stage multimodal fusion framework to preserve modality-specific knowledge as well as take advantage of the complementarity between different modalities. Instead of directly projecting different modalities into a unified space, our multimodal fusion module limits the representations of different modalities independent while leverages bilinear pooling for fusion and incorporates contrastive learning as additional constraints. Furthermore, the decision fusion module delivers the learned weighted average over the predictions of all modalities to better incorporate the complementarity of different modalities. Our approach has been demonstrated to be effective through empirical evaluations on several real-world datasets. The implementation code is available online at https://github.com/HestiaSky/IMF-Pytorch.

Related Papers

• → Leveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection(2022)69 cited
• → Factors Influencing Modality Choice in Multimodal Applications(2008)15 cited
• The Mismatch of Modalities and Its Effects(2012)
• → Study of communication modalities for teaching distance information(2022)
• → TYPES OF MODALITY AND ITS INCONSISTENCIES/ԵՂԱՆԱԿԱՎՈՐՄԱՆ ՏԵՍԱԿՆԵՐԸ ԵՎ ԴՐԱ ՀԵՏ ԿԱՊՎԱԾ ԱՆՀԱՄԱՊԱՏԱՍԽԱՆՈՒԹՅՈՒՆՆԵՐԸ/ТИПЫ МОДАЛЬНОСТИ И ЕЕ НЕСООТВЕТСТВИЯ(2022)