Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition

Citations Over TimeTop 10% of 2025 papers

Abstract

The rising prevalence of skin lesions places a heavy burden on global health resources and necessitates an early and precise diagnosis for successful treatment. The diagnostic potential of recent multi-modal skin lesion detection algorithms is limited because they ignore dynamic interactions and information sharing across modalities at various feature scales. To address this, we propose a deep learning framework, Multi-Modal Skin-Imaging-based Information-Switching Network (MDSIS-Net), for end-to-end skin lesion recognition. MDSIS-Net extracts intra-modality features using transfer learning in a multi-scale fully shared convolutional neural network and introduces an innovative information-switching module. A cross-attention mechanism dynamically calibrates and integrates features across modalities to improve inter-modality associations and feature representation in this module. MDSIS-Net is tested on clinical disfiguring dermatosis data and the public Derm7pt melanoma dataset. A Visually Intelligent System for Image Analysis (VISIA) captures five modalities: spots, red marks, ultraviolet (UV) spots, porphyrins, and brown spots for disfiguring dermatosis. The model performs better than existing approaches with an mAP of 0.967, accuracy of 0.960, precision of 0.935, recall of 0.960, and f1-score of 0.947. Using clinical and dermoscopic pictures from the Derm7pt dataset, MDSIS-Net outperforms current benchmarks for melanoma, with an mAP of 0.877, accuracy of 0.907, precision of 0.911, recall of 0.815, and f1-score of 0.851. The model's interpretability is proven by Grad-CAM heatmaps correlating with clinical diagnostic focus areas. In conclusion, our deep multi-modal information-switching model enhances skin lesion identification by capturing relationship features and fine-grained details across multi-modal images, improving both accuracy and interpretability. This work advances clinical decision making and lays a foundation for future developments in skin lesion diagnosis and treatment.

Related Papers

• → Leveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection(2022)69 cited
• → Factors Influencing Modality Choice in Multimodal Applications(2008)15 cited
• The Mismatch of Modalities and Its Effects(2012)
• → Study of communication modalities for teaching distance information(2022)
• → TYPES OF MODALITY AND ITS INCONSISTENCIES/ԵՂԱՆԱԿԱՎՈՐՄԱՆ ՏԵՍԱԿՆԵՐԸ ԵՎ ԴՐԱ ՀԵՏ ԿԱՊՎԱԾ ԱՆՀԱՄԱՊԱՏԱՍԽԱՆՈՒԹՅՈՒՆՆԵՐԸ/ТИПЫ МОДАЛЬНОСТИ И ЕЕ НЕСООТВЕТСТВИЯ(2022)