LongT5: Efficient Text-To-Text Transformer for Long Sequences

Citations Over TimeTop 1% of 2022 papers

Abstract

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present LongT5, a new model that explores the effects of scaling both the input length and model size at the same time. Specifically, we integrate attention ideas from long-input transformers (ETC), and adopt pretraining strategies from summarization pretraining (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call Transient Global (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-ofthe-art results on several summarization and question answering tasks, as well as outperform the original T5 models on these tasks. We have open sourced our architecture and training code, as well as our pre-trained model checkpoints.

Related Papers

• → Improving the BERT Model with Proposed Named Entity Recognition Method for Question Answering(2021)7 cited
• → Abstractive Turkish Text Summarization and Cross-Lingual Summarization Using Transformer(2022)1 cited
• → AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization(2022)6 cited
• → mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences(2023)1 cited
• A Survey on Paraphrasing Technology(2006)