Damped Newton Stochastic Gradient Descent Method for Neural Networks Training

Citations Over TimeTop 14% of 2021 papers

Abstract

First-order methods such as stochastic gradient descent (SGD) have recently become popular optimization methods to train deep neural networks (DNNs) for good generalization; however, they need a long training time. Second-order methods which can lower the training time are scarcely used on account of their overpriced computing cost to obtain the second-order information. Thus, many works have approximated the Hessian matrix to cut the cost of computing while the approximate Hessian matrix has large deviation. In this paper, we explore the convexity of the Hessian matrix of partial parameters and propose the damped Newton stochastic gradient descent (DN-SGD) method and stochastic gradient descent damped Newton (SGD-DN) method to train DNNs for regression problems with mean square error (MSE) and classification problems with cross-entropy loss (CEL). In contrast to other second-order methods for estimating the Hessian matrix of all parameters, our methods only accurately compute a small part of the parameters, which greatly reduces the computational cost and makes the convergence of the learning process much faster and more accurate than SGD and Adagrad. Several numerical experiments on real datasets were performed to verify the effectiveness of our methods for regression and classification problems.

Related Papers

• → Newton-SOR Iteration for Solving Large-Scale Unconstrained Optimization Problems with an Arrowhead Hessian Matrices(2019)7 cited
• → Quasi-Newton Methods(2019)7 cited
• → Maximum entropy derivation of quasi-Newton methods(2016)1 cited
• → Newton-2EGSOR Method for Unconstrained Optimization Problems with a Block Diagonal Hessian(2019)1 cited
• → Diagonal quasi-Newton updating formula using log-determinant norm(2016)