Balanced Importance Sampling Estimation

In this paper we analyze a particular issue of estimation, namely the estimation of the expected value of an unknown function for a given distribution, with the samples drawn from other distributions. A motivation of this problem comes from machine learning. In reinforcement learning, an intelligent agent that learns to make decisions in an unknown environment encounters the problem of judging an arbitrary decision policy (the given distribution) on the basis of previous decisions and their outcomes suggested by previous policies (other distributions). The problem can be solved with the use of well established importance sampling estimators. To overcome a potential problem of excessive variance of such estimators, we introduce the family of balanced importance sampling estimators, prove their consistency and demonstrate empirically their superiority over the classical counterparts.