The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Citations Over Time
Abstract
Recurrent nets are in principle capable to store past inputs to produce the currently desired output. Because of this property recurrent nets are used in time series prediction and process control. Practical applications involve temporal dependencies spanning many time steps, e.g. between relevant inputs and desired outputs. In this case, however, gradient based learning methods take too much time. The extremely increased learning time arises because the error vanishes as it gets propagated back. In this article the de-caying error flow is theoretically analyzed. Then methods trying to overcome vanishing gradients are briefly discussed. Finally, experiments comparing conventional algorithms and alternative methods are presented. With advanced methods long time lag problems can be solved in reasonable time.
Related Papers
- → "Investigating the relationship between “cited time lag” and “citing time lag”: A journal study in Information Science"(2023)1 cited
- Review of Time-lag in Knowledge Spillovers(2013)
- Time-Lag Effect of Urban Land Market Regulations:A Case of Wuhan City(2010)
- → Deflection Lag, Load Lag, and Time Lag of Buried Flexible Pipe(2011)
- → What types of challenges do countries face in addressing curriculum time lag, and what strategies do they use to address these challenges?(2020)