Predictions with Con dence Intervals ( Local Error Bars )

Predictions with Con dence Intervals (Local Error Bars) Andreas S. Weigend Department of Computer Science and Institute of Cognitive Science University of Colorado Boulder, CO 80309-0430 [email protected] David A. Nix Department of Computer Science and Institute of Cognitive Science University of Colorado Boulder, CO 80309-0430 [email protected] Abstract|We present a new method for obtaining local error bars, i.e., estimates of the con dence in the predicted value that depend on the input. We approach this problem of nonlinear regression in a maximum likelihood framework. We demonstrate our technique rst on computer generated data with locally varying, normally distributed target noise. We then apply it to the laser data from the Santa Fe Time Series Competition. Finally, we extend the technique to estimate error bars for iterated predictions, and apply it to the exact competition task where it gives the best performance to date. 1 Obtaining Error Bars Using a Maximum Likelihood Framework 1.1 Motivation and Concept Feed-forward arti cial neural networks are widely used and well-suited for nonlinear regression. They can be interpreted as predicting the expected value of the conditional target distribution as a function of (or \conditioned on") the input pattern (e.g., Buntine & Weigend, 1991). This target distribution in response to each input may also be viewed as an error model (Rumelhart et al., 1994). Often an estimate of the mean of this conditional target distribution su ces; this is typically done using one output unit. However, we here present a method that gives more information than just the mean of that distribution. Such additional information could be obtained by attempting to estimate the entire conditional target distribution with connectionist methods (e.g., \fractional binning," Srivastava & Weigend, 1994) or with non-connectionist methods such as a Monte Carlo on a Hidden Markov Model (Fraser & Dimitriadis, 1994). Nonparametric estimates of the shape of a conditional target distribution require large quantities of data. In contrast, our less data-hungry method assumes a speci c parameterized form of the conditional target distribution (e.g., Gaussian) and gives us the value of the error bar (e.g., the width of the Gaussian) by nding those parameters that maximize the likelihood of the data given the model. Speci cally, when we use a network output y to approximate a function f(x), we assume that the desired output (d) (i.e., the observed values) can be modeled by d(x) = f(x) + n(x) where n(x) is noise drawn from the assumed error model distribution. Just as the estimate of the mean of this distribution y(x) is a function of the input, the variance 2 may also vary as a function of the location in input space x. When the noise level varies over the input space (i.e., 2(x) depends on x; it is not a constant), not only do we want the network to learn an output function y(x) that estimates the expectation value (x) of the conditional target distribution, but we also want to learn a function v(x) that estimates the variance 2(x) of that distribution. Therefore, we simply add an auxiliary output unit, the v-unit, that computes v(x), our estimate of 2(x). Since 2(x) must be positive, we choose an exponential activation function for v(x) to naturally impose this bound: v(x) = exp [Pk wvkhk(x) + ], where is the o set (or \bias"), and wvk is the weight between hidden unit k and the v-unit. This architecture is sketched in Figure 1.