Loss-Calibrated Monte Carlo Action Selection

Bayesian decision-theory underpins robust decisionmaking in applications ranging from plant control to robotics where hedging action selection against state uncertainty is critical for minimizing low probability but potentially catastrophic outcomes (e.g, uncontrollable plant conditions or robots falling into stairwells). Unfortunately, belief state distributions in such settings are often complex and/or high dimensional, thus prohibiting the efficient application of analytical techniques for expected utility computation when real-time control is required. This leaves Monte Carlo evaluation as one of the few viable (and hence frequently used) techniques for online action selection. However, loss-insensitive Monte Carlo methods may require large numbers of samples to identify optimal actions with high certainty since they may sample from high probability regions that do not disambiguate action utilities. In this paper we remedy this problem by deriving an optimal proposal distribution for a loss-calibrated Monte Carlo importance sampler that bounds the regret of using an estimated optimal action. Empirically, we show that using our loss-calibrated Monte Carlo method yields high-accuracy optimal action selections in a fraction of the number of samples required by conventional loss-insensitive samplers. Introduction Bayesian decision-theory (Gelman et al. 1995; Robert 2001; Berger 2010) provides a formalization of robust decisionmaking in uncertain settings by maximizing expected utility. Formally, a utility function u(θ, a) quantifies the return of performing an action a ∈ A = {a1, . . . , ak} in a given state θ. When the true state is uncertain and only a belief state distribution p(θ) is known, Bayesian decision-theory posits that an optimal control action a should maximize the expected utility (EU) U(a) = E[u(θ, a)] = ˆ u(θ, a)p(θ)dθ. (1) where by definition, the optimal action a∗ is a∗ = arg max a U(a). (2) In real-world settings such as robotics (Thrun 2000), the Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. a1