Improving Safety in Deep Reinforcement Learning using Unsupervised Action Planning

Citations Over Time

Abstract

One of the key challenges to deep reinforcement learning (deep RL) is to ensure safety at both training and testing phases. In this work, we propose a novel technique of unsupervised action planning to improve the safety of on-policy reinforcement learning algorithms, such as trust region policy optimization (TRPO) or proximal policy optimization (PPO). We design our safety-aware reinforcement learning by storing all the history of “recovery” actions that rescue the agent from dangerous situations into a separate “safety” buffer and finding the best recovery action when the agent encounters similar states. Because this functionality requires the algorithm to query similar states, we implement the proposed safety mechanism using an unsupervised learning algorithm, k-means clustering. We evaluate the proposed algorithm on six robotic control tasks that cover navigation and manipulation. Our results show that the proposed safe RL algorithm can achieve higher rewards compared with multiple baselines in both discrete and continuous control problems. The supplemental video can be found at: https://youtu.be/AFTeWSohILo.

Related Papers

• → Advances in Machine Learning and Deep Learning Algorithms and Their Applications(2021)1 cited
• → Survey of Machine Learning Algorithms & its Applications(2021)1 cited
• → The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning(2020)17 cited
• Tidal Wave Prediction Using Machine Learning(2021)
• → The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons\n from Infant Learning(2020)