Self-Training With Noisy Student Improves ImageNet Classification

Citations Over TimeTop 1% of 2020 papers

Abstract

We present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. To achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher.

Related Papers

• → MOOC Dropout Prediction(2017)86 cited
• → Controlled dropout: A different approach to using dropout on deep neural network(2017)38 cited
• → A model for the identification of students at risk of dropout at a university of technology(2020)18 cited
• → Using Stacked Denoising Autoencoder for the Student Dropout Prediction(2017)13 cited
• Exploration of High Rate of Dropout in Distance Educaion(2007)