Incorporating Pre-trained Model into Neural Machine Translation

Citations Over Time

Abstract

Since Transformer is proposed, it has boosted the neural machine translation baseline. In the past few years, although many researchers have tried to update the original transformer for each module, these approaches gain a little improvement. Pre-trained models have proved their power in natural language understanding tasks, how to incorporate the pre-trained model into natural language generation tasks such as machine translation is still an opening problem. This paper proposes a novel framework call Transformer-Pt,which combines the BERT and GPT-2 with Transformer. Experiment result on the WMT-14 dataset demonstrates our proposed method achieve improvement compared to other models.

Related Papers

• → Training Deeper Neural Machine Translation Models with Transparent Attention(2018)144 cited
• → Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder(2019)65 cited
• → Searching Better Architectures for Neural Machine Translation(2020)27 cited
• → Incorporating Pre-trained Model into Neural Machine Translation(2021)2 cited