A Thorough Evaluation of Task-Specific Pretraining for Summarization
Citations Over TimeTop 10% of 2021 papers
Abstract
Task-agnostic pretraining objectives like masked language models or corrupted span prediction are applicable to a wide range of NLP downstream tasks We compare three summarization specific pretraining objectives with the task agnostic corrupted span prediction pretraining in a controlled study. We also extend our study to a low resource and zero shot setup, to understand how many training examples are needed in order to ablate the task-specific pretraining without quality loss. Our results show that task-agnostic pretraining is sufficient for most cases which hopefully reduces the need for costly task-specific pretraining. We also report new state-of-the-art number for two summarization tasks using a T5 model with 11 billion parameters and an optimal beam search length penalty.
Related Papers
- Behavior-Based Segmentation of Demonstrated Task(2006)
- → The Application of Task-Driven Teaching Method(2011)5 cited
- → Task recognition based on task history considering estimation error of localization in care support with autonomous mobile wheelchairs(2020)3 cited
- Task Patterns for Human-Robot Interaction(2002)
- → Co-development of task models through robot-human interaction(2007)