The MultiBERTs: BERT Reproductions for Robustness Analysis

Citations Over Time

Abstract

Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function. Recent work has shown that repeating the pre-training process can lead to substantially different performance, suggesting that an alternate strategy is needed to make principled statements about procedures. To enable researchers to draw more robust conclusions, we introduce the MultiBERTs, a set of 25 BERT-Base checkpoints, trained with similar hyper-parameters as the original BERT model but differing in random weight initialization and shuffling of training data. We also define the Multi-Bootstrap, a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data. To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint. We release our models and statistical library along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.

Related Papers

• → (2019)31,543 cited
• → HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case(2019)17,209 cited
• → Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank(2013)6,670 cited
• → Rethinking the Objectives of Extractive Question Answering(2021)3 cited