Learning long-term dependencies with gradient descent is difficult
IEEE Transactions on Neural Networks1994Vol. 5(2), pp. 157–166
Citations Over TimeTop 1% of 1994 papers
Abstract
Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered.
Related Papers
- → Stochastic Gradient Descent(2015)47 cited
- → Deep Recurrent Neural Network (Deep-RNN) for Classification of Nonlinear Data(2020)12 cited
- → Training a Two-Layer ReLU Network Analytically(2023)7 cited
- → Accelerating Extreme Search Based on Natural Gradient Descent with Beta Distribution(2021)4 cited
- → Training Neural Networks Using Predictor-Corrector Gradient Descent(2018)