Learning Pessimism for Reinforcement Learning

Off-policy deep reinforcement learning algorithms commonly compensate for overestimation bias during temporal-difference by utilizing pessimistic estimates of the expected target returns. In this work, we propose Generalized Pessimism Learning (GPL), a strategy employing novel learnable penalty to enact such pessimism. particular, learn alongside critic with dual TD-learning, new procedure estimate and minimize magnitude returns trivial computational cost. GPL enables us accurately counteract throughout training without incurring downsides overly targets. By integrating popular off-policy algorithms, achieve state-of-the-art results in both competitive proprioceptive pixel-based benchmarks.