TOT：Topology-Aware Optimal Transport for Multimodal Hate Detection

Citations Over TimeTop 13% of 2023 papers

Abstract

Multimodal hate detection, which aims to identify the harmful content online such as memes, is crucial for building a wholesome internet environment. Previous work has made enlightening exploration in detecting explicit hate remarks. However, most of their approaches neglect the analysis of implicit harm, which is particularly challenging as explicit text markers and demographic visual cues are often twisted or missing. The leveraged cross-modal attention mechanisms also suffer from the distributional modality gap and lack logical interpretability. To address these semantic gap issues, we propose TOT: a topology-aware optimal transport framework to decipher the implicit harm in memes scenario, which formulates the cross-modal aligning problem as solutions for optimal transportation plans. Specifically, we leverage an optimal transport kernel method to capture complementary information from multiple modalities. The kernel embedding provides a non-linear transformation ability to reproduce a kernel Hilbert space (RKHS), which reflects significance for eliminating the distributional modality gap. Moreover, we perceive the topology information based on aligned representations to conduct bipartite graph path reasoning. The newly achieved state-of-the-art performance on two publicly available benchmark datasets, together with further visual analysis, demonstrate the superiority of TOT in capturing implicit cross-modal alignment.

Related Papers

• → Interruptions as multimodal outputs: which are the less disruptive?(2003)69 cited
• → Beyond a Generic Complementary and Alternative Medicine: The Holistic Health Care- Conventional Medicine Continuum(2009)8 cited
• → Some Issues on Choices of Modalities for Multimodal Biometric Systems(2014)1 cited
• → The Effect of Teaching Practical Physical Modalities on the Ordering Skills of Physical Medicine and Rehabilitation Residents(2013)
• → SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities(2022)