Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets

Citations Over TimeTop 1% of 2013 papers

Abstract

While Deep Neural Networks (DNNs) have achieved tremendous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10–50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and language modeling. We show on three different LVCSR tasks ranging between 50–400 hrs, that a low-rank factorization reduces the number of parameters of the network by 30–50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation.

Related Papers

• → Hard scale uncertainty in collinear factorization: Perspective from kt-factorization(2018)2 cited
• → Partial restoration of factorization and universality in presence of factorization breaking interactions in hadronic hard scattering processes(2007)
• → QCD factorization for the DIS structure functions at small x(2014)
• → Which scale for parton densities? Highlight from $k_{t}$-factorization(2018)