A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

Citations Over TimeTop 1% of 2020 papers

Abstract

Thus far, end-to-end (E2E) models have not been shown to outperform state-of-the-art conventional models with respect to both quality, i.e., word error rate (WER), and latency, i.e., the time the hypothesis is finalized after the user stops speaking. In this paper, we develop a first-pass Recurrent Neural Network Transducer (RNN-T) model and a second-pass Listen, Attend, Spell (LAS) rescorer that surpasses a conventional model in both quality and latency. On the quality side, we incorporate a large number of utterances across varied domains [1] to increase acoustic diversity and the vocabulary seen by the model. We also train with accented English speech to make the model more robust to different pronunciations. In addition, given the increased amount of training data, we explore a varied learning rate schedule. On the latency front, we explore using the end-of-sentence decision emitted by the RNN-T model to close the microphone, and also introduce various optimizations to improve the speed of LAS rescoring. Overall, we find that RNN-T+LAS offers a better WER and latency tradeoff compared to a conventional model. For example, for the same latency, RNN-T+LAS obtains a 8% relative improvement in WER, while being more than 400-times smaller in model size.

Related Papers

• → A Spelling Correction Model for End-to-end Speech Recognition(2019)140 cited
• → Bidirectional recurrent neural network language models for automatic speech recognition(2015)83 cited
• → Language Models with RNNs for Rescoring Hypotheses of Russian ASR(2016)3 cited
• → Large Scale Language Modeling in Automatic Speech Recognition(2012)37 cited
• → A spelling correction model for end-to-end speech recognition(2019)2 cited