TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Citations Over TimeTop 1% of 2023 papers

Abstract

The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on the transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.

Related Papers

• → Comparative analysis of different fusion rules for SAR and multi-spectral image fusion based on NSCT and IHS transform(2015)9 cited
• → A Novel Fusion Rule for Medical Image Fusion in Complex Wavelet Transform Domain(2016)7 cited
• → Better and Faster Deep Image Fusion with Spatial Frequency(2021)1 cited
• Image fusion algorithm based on high frequency channel(2003)
• → Cognition-based Fusion Method for Infrared and Visible image(2018)