From Strings to Things: Knowledge-Enabled VQA Model That Can Read and Reason
Citations Over TimeTop 10% of 2019 papers
Abstract
Text present in images are not merely strings, they provide useful cues about the image. Despite their utility in better image understanding, scene texts are not used in traditional visual question answering (VQA) models. In this work, we present a VQA model which can read scene texts and perform reasoning on a knowledge graph to arrive at an accurate answer. Our proposed model has three mutually interacting modules: i. proposal module to get word and visual content proposals from the image, ii. fusion module to fuse these proposals, question and knowledge base to mine relevant facts, and represent these facts as multi-relational graph, iii. reasoning module to perform a novel gated graph neural network based reasoning on this graph. The performance of our knowledge-enabled VQA model is evaluated on our newly introduced dataset, viz. text-KVQA. To the best of our knowledge, this is the first dataset which identifies the need for bridging text recognition with knowledge graph based reasoning. Through extensive experiments, we show that our proposed method outperforms traditional VQA as well as question-answering over knowledge base-based methods on text-KVQA.
Related Papers
- → Choice-Driven Contextual Reasoning for Commonsense Question Answering(2022)3 cited
- → Knowledge Base Question Answering for Intelligent Maintenance of Power Plants(2021)3 cited
- → Research on the method of knowledge base question answering(2021)2 cited
- Overview and Methods of Knowledge Base Question Answering(2021)
- → How Additional Knowledge can Improve Natural Language Commonsense Question Answering?(2019)32 cited