A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
arXiv (Cornell University)2015
Citations Over Time
Abstract
Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.
Related Papers
- → A Benchmark Characterization of the EEMBC Benchmark Suite(2009)218 cited
- → Solutions to the Third Benchmark Control Problem(1991)3 cited
- Theoretical Analysis of the Benchmark for Choosing Manipulative Instruments of Monetary Policies(2009)
- → Exploring disk performance benchmarks(2017)
- → Support Structure Performance Benchmark(2023)