MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
arXiv (Cornell University)2019
Citations Over Time
Julian Martin Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, Jeremy Howard
Abstract
Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code.
Related Papers
- → (2019)31,543 cited
- → MultiFiT: Efficient Multi-lingual Language Model Fine-tuning(2019)93 cited
- → Data Augmentation with Adversarial Training for Cross-Lingual NLI(2021)12 cited
- → Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning(2020)6 cited
- → Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification(2017)