An Empirical Study of Example Forgetting during Deep Neural Network Learning

Citations Over Time

Abstract

Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

Related Papers

• → Deep Residual Learning for Image Recognition(2016)216,943 cited
• → ImageNet: A large-scale hierarchical image database(2009)60,619 cited
• → (2019)31,543 cited
• → Learning Multiple Layers of Features from Tiny Images(2024)25,445 cited
• → An Empirical Study of Example Forgetting during Deep Neural Network\n Learning(2018)199 cited