0 citations0 references

Addressing Hindsight Bias in Multigoal Reinforcement Learning

IEEE Transactions on Cybernetics2021Vol. 53(1), pp. 392–405

Citations Over TimeTop 13% of 2021 papers

Chenjia Bai, Lingxiao Wang, Zhaoran Wang, Zhaoran Wang, Rui Zhao, Chenyao Bai, Peng Liu

Abstract

Multigoal reinforcement learning (RL) extends the typical RL with goal-conditional value functions and policies. One efficient multigoal RL algorithm is the hindsight experience replay (HER). By treating a hindsight goal from failed experiences as the original goal, HER enables the agent to receive rewards frequently. However, a key assumption of HER is that the hindsight goals do not change the likelihood of the sampled transitions and trajectories used in training, which is not the fact according to our analysis. More specifically, we show that using hindsight goals changes such a likelihood and results in a biased learning objective for multigoal RL. We analyze the hindsight bias due to this use of hindsight goals and propose the bias-corrected HER (BHER), an efficient algorithm that corrects the hindsight bias in training. We further show that BHER outperforms several state-of-the-art multigoal RL approaches in challenging robotics tasks.

Related Papers

→ Addressing Hindsight Bias in Multigoal Reinforcement Learning(2021)18 cited
→ Domain Knowledge and Hindsight Bias among Poker Players(2013)8 cited
Motivational influences on the strength of the hindsight bias(2002)
→ Supplemental Material for Is There Hindsight Bias Without Real Hindsight? Conjectures Are Sufficient to Elicit Hindsight Bias(2018)