Dynamics of Temporal Difference Learning

In behavioural sciences, the problem that a sequence of stimuli is followed by a sequence of rewards r(t) is considered. The subject is to learn the full sequence of rewards from the stimuli, where the prediction is modelled by the Sutton-Barto rule. In a sequence of n trials, this prediction rule is learned iteratively by temporal difference learning. We present a closed formula of the prediction of rewards at trial time t within trial n. From that formula, we show directly that for n → ∞ the predictions converge to the real rewards. In this approach, a new quality of correlation type Toeplitz matrices is proven. We give learning rates which optimally speed up the learning process. 1 Temporal Difference Learning We consider here a mathematical treatment of a problem in behavioural biology. This problem has been described e.g. in [Dayan and Abbott 2001] as learning to predict a reward. It is Pavlovian in the sense that classical conditioning is adressed, however reward is not immediate, but after a series of stimuli there is a latency time, followed by a series of rewards. Using a simple linear rule, we will show that the subject is able to predict the remaining rewards by that rule, repeating the same stimuli-reward pattern over a series of trials. A review of learning theory related to these problems can be found in [Sutton and Barto 1998]. There are recent experimental biological correlates with this mathematical model, e.g. in the activity of primate dopamine cells during appetitive conditioning tasks, together with the psychological and pharmacological rationale for studying these cells. A review was given in [Schultz 1998], the connection to temporal difference learning can be found in [Montague et al. 1996]. In this paper, the focus is on two mathematical issues: a) to provide a direct constructive proof of convergence by giving an explicit dependence of the prediction error over trials, b) to minimize the learning time by giving a formula for optimal setting of the learning rate. Hence, the paper contributes as well to temporal difference learning as a purely mathematical issue, which may be valuable also without reference to behavioural biology and reinforcement learning. It can also be understood in a Dynamic Programming sense, see [Watkins 1989] and later work, e.g. [Gordon 2001] and [Szepesvari and Smart 2004]. We adopt the following notation in the course of this paper, following [Dayan and Abbott 2001]: • Stimulus u(t) • Future reward r(t) • Sum of future rewards R(t) • Weights w(k) • Predicted reward v(t) We want to compute, for all trials n of duration T , and for any time of trial t, the predicted reward v(t). The (extended) stimulus u(t) is given at times tu,min . . . tu,max , the (extended) reward r(t) is presented at times tr,min . . . tr,max. Stimulus and reward do not overlap, i.e. tr,min > tu,max. The subject is to learn the total remaining reward at time t,