Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
arXiv (Cornell University)2020
Citations Over Time
Ashley D. Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles L. Isbell, Jason Yosinski
Abstract
In this paper, we introduce a novel form of value function, $Q(s, s')$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s'$ and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at http://sites.google.com/view/qss-paper.
Related Papers
- → On Value Function Representation of Long Horizon Problems(2018)19 cited
- → Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimate(2021)7 cited
- Deterministic policy gradient algorithms(2014)
- → On-Line Control Methods via Simulation(2013)
- → An Incremental Fast Policy Search Using a Single Sample Path(2017)