INTEGRATING DECISION-THEORETIC PLANNING AND PROGRAMMING FOR ROBOT CONTROL IN HIGHLY DYNAMIC DOMAINS von

MDPs Yet, the application of options/macros has only been discussed by intuition. One of the models of usage proposed in [21] is the following: Definition 3.2.3 Let Π = {S1, . . . , Sn} be a decomposition of MDP M = 〈A,S,Tr ,R〉, and let A = {Ai : i ≤ n} be a collection of macroaction sets, where Ai = {π1 i , . . . , πi i } is a set of macros for region Si. The abstract MDP M ′ = 〈A′, S′,Tr ′,R′〉 induced by Π and A, is given by: • S′ = PerΠ(S) = ⋃ i≤nEPer(Si) • A′ = iAi with π i ∈ Ai feasible only at states s ∈ EPer(Si) • T ′(s, π i , s′) is given by the discounted transition model for π i , for any s ∈ EPer(Si) and s′ ∈ XPer(Si); T ′(s, π i , s′) = 0 for any s′ 6∈ XPer(Si) • R′(s, π i ) is given by the discounted reward model for π i , for any s ∈ EPer(Si). This seizes our intuition about the general idea of options: We abstract from the original state space to a much smaller one, namely the set of peripheral states. These build some kind of interfaces between the regions. The actions used for planning are the options (macros) defined for the different regions. The models, transition model and reward function, are the discounted models that have been presented. Recall that these form the crucial part in planning at the level of options. We point out that this reduction of complexity, which will finally speed up computation as we will see, comes at the cost of possibly finding only a sub-optimal solution. The examples in [21] have been conducted in a grid world which is depicted in Figure 3.6. This navigation task, where negative rewards are to be minimized, was solved 7Alternatively, one can define the local MDP without the additional state α and instead make all exit states absorbing with an empty feasible set. 8[21] also allows for the goal of staying in a region, modeled by low values for all exits. CHAPTER 3. MARKOV DECISION PROCESSES AND OPTIONS 31 (a) Maze121 (b) peripheral states Figure 3.6: Taken from [21]: The example environment for testing the abstract MDP against the original MDP; (a) The agent can move in any compass direction or not move at all. Moving is uncertain: with some small probability the agent may move in another direction than intended. Each cell gives negative reward, except the upper right cell, which is absorbing and thus forms the goal. Shaded squares have a higher negative reward, on patterned squares moving is even more uncertain (probability of failure is higher). Shaded circles denote absorbing states with high negative reward; (b) peripheral states for a decomposition into 11 regions (rooms). with the original MDP as well as with the abstract MDP. Additionally an augmented MDP was tested which we are going to leave out of consideration here. For the abstract MDP the set of macros was created based on the heuristic approach described above – one macro for each region-exit state combination, plus one for staying in the room.9 Value iteration was applied to solve the different MDPs. Figure 3.7 shows the value for one particular state and how it improves over time. Clearly, with the abstract MDP the value function converges much faster. But, recognizable from the limit of the value function, the abstract MDP finds only a suboptimal solution: It finds a policy which takes the agent to the goal with expected costs (negative reward) of over 20, while the original MDP finds a way where less than 20 are expected. Nevertheless, the computational saving seem worth the drawback on solution quality.10