Vision–Language Model for Visual Question Answering in Medical Imagery

Citations Over TimeTop 10% of 2023 papers

Abstract

In the clinical and healthcare domains, medical images play a critical role. A mature medical visual question answering system (VQA) can improve diagnosis by answering clinical questions presented with a medical image. Despite its enormous potential in the healthcare industry and services, this technology is still in its infancy and is far from practical use. This paper introduces an approach based on a transformer encoder-decoder architecture. Specifically, we extract image features using the vision transformer (ViT) model, and we embed the question using a textual encoder transformer. Then, we concatenate the resulting visual and textual representations and feed them into a multi-modal decoder for generating the answer in an autoregressive way. In the experiments, we validate the proposed model on two VQA datasets for radiology images termed VQA-RAD and PathVQA. The model shows promising results compared to existing solutions. It yields closed and open accuracies of 84.99% and 72.97%, respectively, for VQA-RAD, and 83.86% and 62.37%, respectively, for PathVQA. Other metrics such as the BLUE score showing the alignment between the predicted and true answer sentences are also reported.

Related Papers

• → A Comprehensive Review on Deep Learning Approaches for Question Answering and Machine Reading Comprehension in NLP(2023)6 cited
• → Utilizing Bidirectional Encoder Representations from Transformers for Answer Selection(2021)7 cited
• → FNet with Cross-Attention Encoder for Visual Question Answering(2022)1 cited
• → A Question Answer Approach to Building Semantic Memory(2012)
• → A bio- medical Question Answering system for the Malayalam language using word embedding and Bidirectional Encoder Representation from Transformers.(2022)