Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction

Value function is the central notion of Reinforcement Learning (RL). estimation, especially with approximation, can be challenging since it involves stochasticity environmental dynamics and reward signals that sparse delayed in some cases. A typical model-free RL algorithm usually estimates values a policy by Temporal Difference (TD) or Monte Carlo (MC) algorithms directly from rewards, without explicitly taking into consideration. In this paper, we propose Decomposition Future Prediction (VDFP), providing an explicit two-step understanding value estimation process: 1) first foresee latent future, 2) then evaluate it. We analytically decompose future part policy-independent trajectory return part, inducing way to model returns separately estimation. Further, derive practical deep algorithm, consisting convolutional learn compact representation past experiences, conditional variational auto-encoder predict convex evaluates representation. experiments, empirically demonstrate effectiveness our approach for both off-policy on-policy several OpenAI Gym continuous control tasks as well few variants reward.