Aligning machines and minds: neural encoding for high-level visual cortices based on image captioning task

Abstract

Objective.Neural encoding of visual stimuli aims to predict brain responses in the visual cortex to different external inputs. Deep neural networks trained on relatively simple tasks such as image classification have been widely applied in neural encoding studies of early visual areas. However, due to the complex and abstract nature of semantic representations in high-level visual cortices, their encoding performance and interpretability remain limited.Approach.We propose a novel neural encoding model guided by the image captioning task. During image captioning, an attention module is employed to focus on key visual objects. In the neural encoding stage, a flexible receptive field (RF) module is designed to simulate voxel-level visual fields. To bridge the domain gap between these two processes, we introduce the Atten-RF module, which effectively aligns attention-guided visual representations with voxel-wise brain activity patterns.Main results.Experiments on the large-scale natural scenes dataset demonstrate that our method achieves superior average encoding performance across seven high-level visual cortices, with a mean squared error of 0.765, Pearson correlation coefficient of 0.443, and coefficient of determination (R2) of 0.245.Significance.By leveraging the guidance and alignment provided by a complex vision-language task, our model enhances the prediction of voxel activity in high-level visual cortex, offering a new perspective on the neural encoding problem. Furthermore, various visualization techniques provide deeper insights into the neural mechanisms underlying visual information processing.