Stochastic Approximation for Risk-Aware Markov Decision Processes

We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our has two loops. The inner loop computes the risk by solving saddle-point problem. outer performs $Q-$ learning compute an optimal policy. Several widely investigated measures (e.g., conditional value-at-risk, optimized certainty equivalent, and absolute semideviation) are covered our algorithm. Almost sure convergence rate of established. For error tolerance $\epsilon >0$ for $Q$ -value estimation gap $k\in (1/2,\,1]$ , overall is $\Omega ((\ln (1/\delta \epsilon)/\epsilon ^{2})^{1/k}+(\ln (1/\epsilon))^{1/(1-k)})$ with probability at least $1-\delta$ .