Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies
Citations Over Time
Abstract
When we transfer a pretrained language model to a new language, there are many axes of variation that change at once. To disentangle the impact of different factors like syntactic similarity and vocabulary similarity, we propose a set of controlled transfer studies: we systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time, and then measure the resulting drops in a pretrained model's downstream performance. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. %On the other hand, transferring to a dataset with an unaligned vocabulary is extremely hard to recover from in the low-data regime. Moreover, good-quality tokenizers in the transfer language do not make vocabulary alignment easier. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.
Related Papers
- → Reducing Neural Network Parameter Initialization Into an SMT Problem (Student Abstract)(2021)2 cited
- → A New Initialization Method for Neural Networks with Weight Sharing(2021)2 cited
- → Remarks on the initialization of Caputo derivative(2012)4 cited
- The Distributed Initialization Algorithm Based on Known n MSs(2004)
- → Comparison of Random Weight Initialization to New Weight Initialization CONEXP(2020)