The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Citations Over TimeTop 10% of 2021 papers

Abstract

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results 1 .

Related Papers

• → Embedding as a modeling problem(1998)167 cited
• → Behavior therapy: Problems in Generalization(1971)19 cited
• → Chapter 5: The Generalization of Relations and the Dependence of Generalization on Analysis and Abstraction(2021)1 cited
• → A reversible and multiple-layer embedding strategy with high capacity(2010)
• The New Segregation and Generalization of IMO42-2(2005)