Learning small-size DNN with output-distribution-based criteria

Citations Over TimeTop 1% of 2014 papers

Abstract

Deep neural network (DNN) obtains significant accuracy improvements on many speech recognition tasks and its power comes from the deep and wide network structure with a very large number of parameters. It becomes challenging when we deploy DNN on devices which have limited computational and storage resources. The common practice is to train a DNN with a small number of hidden nodes and a small senone set using the standard training process, leading to significant accuracy loss. In this study, we propose to better address these issues by utilizing the DNN output distribution. To learn a DNN with small number of hidden nodes, we minimize the Kullback–Leibler divergence between the output distributions of the small-size DNN and a standard large-size DNN by utilizing a large number of un-transcribed data. For better senone set generation, we cluster the senones in the large set into a small one by directly relating the clustering process to DNN parameters, as opposed to decoupling the senone generation and DNN training process in the standard training. Evaluated on a short message dictation task, the proposed two methods get 5.08% and 1.33% relative word error rate reduction from the standard training method, respectively.

Related Papers

• Study and Two Types of Typical Usage of DataGrid Web Server Control(2005)
• Achieving Parameter of DBSCAN Based on Datagrid(2010)
• Using DataGrid Control to Realize DataBase of Querying in VB6.0(2000)
• Susquehanna Chorale Spring Concert "Roots and Wings"(2017)
• → DETERMINING QUALITY REQUIREMENTS AT THE UNIVERSITIES TO IMPROVE THE QUALITY OF EDUCATION(2018)