DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

Citations Over Time

Abstract

Continual learning aims to enable a single model to learn a sequence of tasks without catastrophic forgetting. Top-performing methods usually require a rehearsal buffer to store past pristine examples for experience replay, which, however, limits their practical value due to privacy and memory constraints. In this work, we present a simple yet effective framework, DualPrompt, which learns a tiny set of parameters, called prompts, to properly instruct a pre-trained model to learn tasks arriving sequentially without buffering past examples. DualPrompt presents a novel approach to attach complementary prompts to the pre-trained backbone, and then formulates the objective as learning task-invariant and task-specific "instructions". With extensive experimental validation, DualPrompt consistently sets state-of-the-art performance under the challenging class-incremental setting. In particular, DualPrompt outperforms recent advanced continual learning methods with relatively large buffer sizes. We also introduce a more challenging benchmark, Split ImageNet-R, to help generalize rehearsal-free continual learning research. Source code is available at https://github.com/google-research/l2p.

Related Papers

• → Ways of Forgetting: Israel and the Obliterated Memory of the Palestinian Nakba(2009)71 cited
• → Measurements of everyday memory: Toward the prevention of forgetting(1984)134 cited
• → Measures of Forgetting(2018)2 cited
• → Selective Forgetting of Deep Networks at a Finer Level than Samples(2020)2 cited
• → The Work of Forgetting(2021)