A Multimodal Variational Encoder-Decoder Framework for Micro-video Popularity Prediction

Citations Over TimeTop 10% of 2020 papers

Abstract

Predicting the popularity of a micro-video is a challenging task, due to a number of factors impacting the distribution such as the diversity of the video content and user interests, complex online interactions, etc. In this paper, we propose a multimodal variational encoder-decoder (MMVED) framework that considers the uncertain factors as the randomness for the mapping from the multimodal features to the popularity. Specifically, the MMVED first encodes features from multiple modalities in the observation space into latent representations and learns their probability distributions based on variational inference, where only relevant features in the input modalities can be extracted into the latent representations. Then, the modality-specific hidden representations are fused through Bayesian reasoning such that the complementary information from all modalities is well utilized. Finally, a temporal decoder implemented as a recurrent neural network is designed to predict the popularity sequence of a certain micro-video. Experiments conducted on a real-world dataset demonstrate the effectiveness of our proposed model in the micro-video popularity prediction task.

Related Papers

• → Found in Translation: Learning Robust Joint Representations by Cyclic Translations between Modalities(2019)411 cited
• → Robust multimodal federated learning for incomplete modalities(2023)16 cited
• → Multimodal Conversational AI: A Survey of Datasets and Approaches(2022)33 cited
• → Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities(2018)38 cited
• → Modality Influence in Multimodal Machine Learning(2023)