Introduction to the Special Issue on Learning Theory
چکیده
This special issue builds on material presented at the Fifteens Annual Conference on Computational Learning Theory (COLT 2002) and the Sixteenth Annual Conference on Neural Information Processing Systems (NIPS*2002), held in Sydney in July 2002 and in Vancouver in December 2002, respectively. We contacted authors who presented work on the analysis of learning in general, and with a particular focus on kernel methods and Bayesian analysis in particular, and asked them to submit an extended journal paper. The six paper in this special issue were selected from a total of nine. Computational learning theory has a long history dating back to the works of Valiant (1984) and Vapnik (1982). While the theory was initially mostly concerned with questions about learnability and sample complexity for specific fixed classes of functions, there has been a recent trend in studying data-dependent function classes and specific learning algorithms. A very powerful class of functions is given by the kernel class (see Schölkopf et al. 1999). Over the last few years, there has been some work on the analysis of the effective complexity of kernel classes based on covering number techniques (Schölkopf et al., 1999, Williamson et al., 2000). In On the Performance of Kernel Classes, Shahar Mendelson presents a series of complexity results for these classes based on localised Rademacher averages. In particular, the paper investigates the ERM algorithm which finds the kernel predictor with the smallest training error. The paper Path Kernels and Multiplicative Updates by Eiji Takimoto and Manfred Warmuth introduces an interesting new class of kernels which can be used directly in online algorithms with multiplicative updates. In a nutshell, path kernels, which are defined by a directed graph, implicitly map each instance (edge labels) to all exponentially many product features defined by all the paths from a designated source to a designates sink node. Using path kernels, the authors are able to recover a large number of known multiplicative update algorithms. The examples include efficient algorithms for learning disjunctions and a recent algorithm that predicts as well as the best pruning of a series of parallel digraphs. Mutliplicative algorithms such as Winnow are known to work robustly in the presence of (exponentially) many irrelevant features (Littlestone, 1988). In his paper Tracking Linear-threshold Concepts with Winnow, Chris Mesterharm gives a slightly modified version of the Winnow algorithm in which demotions cannot reduce the feature weight below a pre-specified threshold ε. This allows him to show that not only is the total number of mistakes bounded by the logarithm of the total number of features but—even stronger—by the logarithm of the length of the longest input vector under the 1–norm. Furthermore, he can show that this algorithm is also able to perform well if the target concepts slightly drifts in between trials. Bayesian approaches to learning have played a significant role in the statistical literature over many years. While they lead to excellent performance in practical applications, there have not been