Deep Residual Networks and Weight Initialization

Citations Over Time

Abstract

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.

Related Papers

• → Reducing Neural Network Parameter Initialization Into an SMT Problem (Student Abstract)(2021)2 cited
• → A New Initialization Method for Neural Networks with Weight Sharing(2021)2 cited
• → Remarks on the initialization of Caputo derivative(2012)4 cited
• The Distributed Initialization Algorithm Based on Known n MSs(2004)
• → Comparison of Random Weight Initialization to New Weight Initialization CONEXP(2020)