Direct and indirect reinforcement learning

Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision-making and control tasks. In this paper, we classify RL into direct indirect according how they seek the optimal policy Markov decision process problem. The former solves by directly maximizing an objective function using gradient descent methods, in which is usually expectation accumulative future rewards. latter indirectly finds solving Bellman equation, sufficient necessary condition from Bellman's principle optimality. We study (PG) forms show that both them can derive actor–critic architecture be unified PG with approximate value stationary state distribution, revealing equivalence RL. employ Gridworld task verify influence different PG, suggesting their differences relationships experimentally. Finally, current mainstream taxonomy, together other ones, including value-based policy-based, model-based model-free.