RealFormer: Transformer Likes Residual Attention

Citations Over TimeTop 10% of 2021 papers

Abstract

Transformer is the backbone of modern NLP models. In this paper, we propose Real-Former, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention.

Related Papers

• → Language Models with Transformers(2019)46 cited
• → Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size(2020)4 cited
• → Finnish Language Modeling with Deep Transformer Models(2020)1 cited