Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder
Citations Over TimeTop 10% of 2019 papers
Abstract
Sharing source and target side vocabularies and word embeddings has been a popular practice in neural machine translation (briefly, NMT) for similar languages (e.g., English to French or German translation). The success of such wordlevel sharing motivates us to move one step further: we consider model-level sharing and tie the whole parts of the encoder and decoder of an NMT model. We share the encoder and decoder of Transformer (Vaswani et al. 2017), the state-of-the-art NMT model, and obtain a compact model named Tied Transformer. Experimental results demonstrate that such a simple method works well for both similar and dissimilar language pairs. We empirically verify our framework for both supervised NMT and unsupervised NMT: we achieve a 35.52 BLEU score on IWSLT 2014 German to English translation, 28.98/29.89 BLEU scores on WMT 2014 English to German translation without/with monolingual data, and a 22.05 BLEU score on WMT 2016 unsupervised German to English translation.
Related Papers
- Pre-Translation for Neural Machine Translation.(2016)
- Better Evaluation Metrics Lead to Better Machine Translation(2011)
- → Improving English-to-Indian Language Neural Machine Translation Systems(2022)31 cited
- → Design and Testing of Automatic Machine Translation System Based on Chinese-English Phrase Translation(2021)7 cited
- Statistical Machine Translation with Rule based Machine Translation.(2011)