Training Deeper Neural Machine Translation Models with Transparent Attention
arXiv (Cornell University)2018
Citations Over Time
Abstract
While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.
Related Papers
- → Neural Machine Translation of Indian Languages(2017)44 cited
- Better Evaluation Metrics Lead to Better Machine Translation(2011)
- → Improving English-to-Indian Language Neural Machine Translation Systems(2022)31 cited
- Statistical Machine Translation with Rule based Machine Translation.(2011)
- → Factored Statistical Machine Translation for German-English(2018)