Verifying the long-range dependency of RNN language models

Citations Over Time

Abstract

It has been argued that recurrent neural network language models are better in capturing long-range dependency than n-gram language models. In this paper, we attempt to verify this claim by investigating the prediction accuracy and the perplexity of these language models as a function of word position, i.e., the position of a word in a sentence. It is expected that as word position increases, the advantage of using recurrent neural network language models over n-gram language models will become more and more evident. On the text corpus of Penn Tree Bank (PTB), a recurrent neural network language model outperforms a trigram language model in both perplexity and word prediction. However, on the AMI meeting corpus, a trigram outperforms a recurrent neural network language model.

Related Papers

• → A dynamic language model for speech recognition(1991)126 cited
• → Class phrase models for language modeling(2002)45 cited
• → Decoupling Word-Pair Distance and Co-occurrence Information for Effective Long History Context Language Modeling(2015)2 cited
• → Improving language models by using distant information(2007)3 cited
• → Verifying the long-range dependency of RNN language models(2016)2 cited