Unsupervised Perceptual Rewards for Imitation Learning

Citations Over TimeTop 1% of 2017 papers

Abstract

Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a reward function takes considerable hand engineering and often requires additional and potentially visible sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple implicit intermediate steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide feedback on these intermediate steps or sub-goals. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit specification of sub-goals. The resulting reward functions, which are dense and smooth, can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward functions, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also demonstrate that our method can be used to learn a complex real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task. Supplementary material and dataset are available at sermanet.github.io/rewards * equal contribution Google Brain Residency program (g.co/brainresidency)

Related Papers

• → Action generation and action perception in imitation: an instance of the ideomotor principle(2003)299 cited
• → Putting the subjective back into intersubjective: The importance of person-specific, distributed, neural representations in perception-action mechanisms(2008)2 cited
• → Thoughts on Imitation as a Perception–Action Coupling: An Essay in Honor of Herbert L. Pick, Jr.(2014)
• Are innovative consumers prone to imitation perception? Developing a constructivist innovativeness model of imitation perception(2016)
• → Analysis of recent advancement in unsupervised deep learning(2022)