Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Citations Over TimeTop 1% of 2020 papers

Abstract

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model's limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the finetuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT's accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.

Related Papers

• → QSTR modelling of the acute toxicity of pharmaceuticals to fish(2012)32 cited
• → The Thirty-one Benchmark Steroids Revisited: Comparative Molecular Moment Analysis (CoMMA) with Principal Component Regression(2000)24 cited
• → B3LYP-SVM METHOD FOR THE ESTIMATION OF MOLECULAR ENTHALPIES OF FORMATION(2007)7 cited
• → The effect of data diversity on the performance of deep learning models for predicting early gastric cancer under endoscopy(2022)2 cited
• → Quantitative Structure-Activity Relationships in Carboquinones and Benzodiazepines Using Counter-Propagation Neural Networks(1995)24 cited