Unbiased Methods for Multi-Goal Reinforcement Learning

Citations Over Time

Abstract

In multi-goal reinforcement learning (RL) settings, the reward for each goal is sparse, and located in a small neighborhood of the goal. In large dimension, the probability of reaching a reward vanishes and the agent receives little learning signal. Methods such as Hindsight Experience Replay (HER) tackle this issue by also learning from realized but unplanned-for goals. But HER is known to introduce bias, and can converge to low-return policies by overestimating chancy outcomes. First, we vindicate HER by proving that it is actually unbiased in deterministic environments, such as many optimal control settings. Next, for stochastic environments in continuous spaces, we tackle sparse rewards by directly taking the infinitely sparse reward limit. We fully formalize the problem of multi-goal RL with infinitely sparse Dirac rewards at each goal. We introduce unbiased deep Q-learning and actor-critic algorithms that can handle such infinitely sparse rewards, and test them in toy environments.

Related Papers

• → Review on Reinforcement-concrete Bonded Anchorage(2022)1 cited
• → Variable Ratio Reinforcement and Differential Reinforcement(1974)
• Susquehanna Chorale Spring Concert "Roots and Wings"(2017)
• → The Effect of Vicarious Reinforcement on Inappropriate Behavior in an Elementary School Classroom(1975)
• → ИСПОЛЬЗОВAНИЕ ПОТЕНЦИAЛA СОЦИAЛЬНЫХ ПAРТНЕРОВ В ПОДГОТОВКЕ БУДУЩИХ ПЕДAГОГОВ(2024)