Direct Neural Machine Translation with Task-level Mixture of Experts models

Abstract

Direct neural machine translation (direct NMT) is a type of NMT system that translates text between two non-English languages. Direct NMT systems often face limitations due to the scarcity of parallel data between non-English language pairs. Several approaches have been proposed to address this limitation, such as multilingual NMT and pivot NMT (translation between two languages via English). Task-level Mixture of expert models (Task-level MoE), an inference-efficient variation of Transformer-based models, has shown promising NMT performance for a large number of language pairs. In Task-level MoE, different language groups can use different routing strategies to optimize cross-lingual learning and inference speed. In this work, we examine Task-level MoE's applicability in direct NMT and propose a series of high-performing training and evaluation configurations, through which Task-level MoE-based direct NMT systems outperform bilingual and pivot-based models for a large number of low and high-resource direct pairs, and translation directions. Our Task-level MoE with 16 experts outperforms bilingual NMT, Pivot NMT models for 7 language pairs, while pivot-based models still performed better in 9 pairs and directions.

Related Papers

• → Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder(2019)65 cited
• → Neural Machine Translation with the Transformer and Multi-Source Romance Languages for the Biomedical WMT 2018 task(2018)15 cited
• → Incorporating Pre-trained Model into Neural Machine Translation(2021)2 cited
• → Recurrent Stacking of Layers for Compact Neural Machine Translation Models(2018)2 cited
• → On The Alignment Problem In Multi-Head Attention-Based Neural Machine\n Translation(2018)