NEAT for large-scale reinforcement learning through evolutionary feature learning and policy gradient search

Citations Over TimeTop 19% of 2018 papers

Abstract

NeuroEvolution of Augmenting Topology (NEAT) is one of the most successful algorithms for solving traditional reinforcement learning (RL) tasks such as pole-balancing. However, the algorithm faces serious challenges while tackling problems with large state spaces, particularly the Atari game playing tasks. This is due to the major flaw that NEAT aims at evolving a single neural network (NN) that must be able to simultaneously extract high-level state features and select action outputs. However such complicated NNs cannot be easily evolved directly through NEAT. To address this issue, we propose a new reinforcement learning scheme based on NEAT with two key technical advancements: (1) a new three-stage learning scheme is introduced to clearly separate feature learning and policy learning to allow effective knowledge sharing and learning across multiple agents; (2) various policy gradient search algorithms can be seamlessly integrated with NEAT for training policy networks with deep structures to achieve effective and sample efficient RL. Experiments on several Atari games confirm that our new learning scheme can be more effective and has higher sample efficiency than NEAT and three state-of-the-art algorithms from the most recent RL literature.

Related Papers

• → Efficient Non-linear Control Through Neuroevolution(2006)126 cited
• → Incorporating Advice into Neuroevolution of Adaptive Agents(2006)5 cited
• Incorporating Advice into Evolution of Neural Networks(2005)
• → Neuroevolution in Games: State of the Art and Open Challenges(2014)3 cited