Learning Qualitative Markov Decision Processes Learning Qualitative Markov Decision Processes

To navigate in natural environments, a robot must decide the best action to take according to its current situation and goal, a problem that can be represented as a Markov Decision Process (MDP). In general, it is assumed that a reasonable state representation and transition model can be provided by the user to the system. When dealing with complex domains, however, it is not always easy or possible to provide such information. In this paper, a system is described that can automatically produce a state abstraction and can learn a transition function over such abstracted states, called q-states. A qualitative state is a group of states with similar properties and rewards. They are induced from the reward function using decision trees. The transition model, represented as a factored MDP, is learned using a Bayesian network learning algorithm. The outcome of this combined learning process produces a very compact MDP that can be efficiently solved using standard techniques. We show experimentally that this approach can learn efficiently a reasonable policy for a mobile robot in large and complex domains.