Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling

Despite the wide applications of Adam in reinforcement learning (RL), theoretical convergence Adam-type RL algorithms has not been established. This paper provides first such analysis for two fundamental policy gradient (PG) and temporal difference (TD) that incorporate AMSGrad updates (a standard alternative analysis), referred to as PG-AMSGrad TD-AMSGrad, respectively. Moreover, our focuses on Markovian sampling both algorithms. We show under general nonlinear function approximation, with a constant stepsize converges neighborhood stationary point at rate O(1/T) (where T denotes number iterations), diminishing exactly O(log^2 T/√T). Furthermore, linear TD-AMSGrad global optimum O(1/T), O(log Our study develops new techniques analyzing sampling.