Training Language Models for Long-Span Cross-Sentence Evaluation
Citations Over TimeTop 10% of 2019 papers
Abstract
While recurrent neural networks can motivate cross-sentence language modeling and its application to automatic speech recognition (ASR), corresponding modifications of the training method for that end are rarely discussed. In fact, even more generally, the impact of training sequence construction strategy in language modeling for different evaluation conditions is typically ignored. In this work, we revisit this basic but fundamental question. We train language models based on long short-term memory recurrent neural networks and Transformers using various types of training sequences and study their robustness with respect to different evaluation modes. Our experiments on 300h Switchboard and Quaero English datasets show that models trained with back-propagation over sequences consisting of concatenation of multiple sentences with state carry-over across sequences effectively outperform those trained with the sentence-level training, both in terms of perplexity and word error rates for cross-utterance ASR.
Related Papers
- Combination of Recurrent Neural Networks and Factored Language Models for Code-Switching Language Modeling(2013)
- → Improved topic-dependent language modeling using information retrieval techniques(1999)55 cited
- → Verifying the long-range dependency of RNN language models(2016)2 cited
- → Going Wider: Recurrent Neural Network With Parallel Cells(2017)5 cited
- → Building Personalized Language Models Through Language Model Interpolation(2023)