A Comprehensive Overhaul of Feature Distillation
Citations Over TimeTop 1% of 2019 papers
Abstract
We investigate the design aspects of feature distillation methods achieving network compression and propose a novel feature distillation method in which the distillation loss is designed to make a synergy among various aspects: teacher transform, student transform, distillation feature position and distance function. Our proposed distillation loss includes a feature transform with a newly designed margin ReLU, a new distillation feature position, and a partial L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2 distance function to skip redundant information giving adverse effects to the compression of student. In ImageNet, our proposed method achieves 21.65% of top-1 error with ResNet50, which outperforms the performance of the teacher network, ResNet152. Our proposed method is evaluated on various tasks such as image classification, object detection and semantic segmentation and achieves a significant performance improvement in all tasks. The code is available at bhheo.github.io/overhaul.
Related Papers
- → Large margin distribution machine(2014)105 cited
- Intensive Margin and Extensive Margin Adjustments of Labor Market : Turkey versus United States(2013)
- → Large Margin Distribution Machine(2013)1 cited
- On the Necessity and Feasibility of Constructing Big Water Margin Cultural System——Taking Big Folk Water Margin by Fan Chaoyang and Zhang Qingjian as an Example(2009)