Multimodal Relation Extraction with Efficient Graph Alignment
Citations Over TimeTop 10% of 2021 papers
Abstract
Relation extraction (RE) is a fundamental process in constructing knowledge graphs. However, previous methods on relation extraction suffer sharp performance decline in short and noisy social media texts due to a lack of contexts. Fortunately, the related visual contents (objects and their relations) in social media posts can supplement the missing semantics and help to extract relations precisely. We introduce the multimodal relation extraction (MRE), a task that identifies textual relations with visual clues. To tackle this problem, we present a large-scale dataset which contains 15000+ sentences with 23 pre-defined relation categories. Considering that the visual relations among objects are corresponding to textual relations, we develop a dual graph alignment method to capture this correlation for better performance. Experimental results demonstrate that visual contents help to identify relations more precisely against the text-only baselines. Besides, our alignment method can find the correlations between vision and language, resulting in better performance. Our dataset and code are available at https://github.com/thecharm/Mega.
Related Papers
- → Review of entity relation extraction(2023)16 cited
- → Information Extraction in the Medical Domain(2015)10 cited
- → Extraction of Spatial Relation in Arabic Text Using Rule-Based Approach(2012)6 cited
- Relation extraction from text documents(2011)
- Simple ontologies for practical information extraction and advanced information extraction for practical ontologies(2013)