Last class, we discussed an analogue for Occam’s Razor for infinite hypothesis spaces that, in conjunction with VC-dimension, reduced the problem of finding a good PAClearning algorithm to the problem of computing the VC-dimension of a given hypothesis space. Recall that VC-dimesion is defined using the notion of a shattered set, i.e. a subset S of the domain such that ΠH(S) = 2 |S|. In this le...

In this first lecture, we begin by introducing the Chinese Restaurant Process. After a brief review of finite mixture models, we describe the Chinese Restaurant Process mixture, where the latent variables are distributed according to a Chinese Restaurant Process. We end by noting a connection between Dirchlet processes and the Chinese Restaurant Process mixture. We will define a distribution on...

The complete nucleotide sequence (5793 bp) of the cosmid vector pTL5 and the origin of its genetic components has been determined. Cosmid pTL5, a derivative of cosmid vector pHC79, is composed of genetic components from pBR322, bacteriophage lambda and the hybrid lambdoid bacteriophage Charon (Ch) 4A cohesive ends (cos) region. The Ch4A cos region contains genetic components from two bacterioph...

The previous lecture defined the Nearest Neighbor Algorithm and discussed how it suffers from the curse of dimensionality. This means that as the number of dimensions increase the Nearest Neighbor algorithm performs poorer and poorer. To better understand the curse of dimensionality with regard to the Nearest Neighbor algorithm one must understand what higher dimensions look like. The following...

The dot sign means inner product. If b is forced to be 0, the VC-dimension reduces to n. It is often the case that the VC-dimension is equal to the number of free parameters of a concept (for example, a rectangle’s parameters are its topmost, bottommost, leftmost and rightmost bounds, and its VC-dimension is 4). However, it is not always true; there exists concepts with 1 parameter but an infin...

These are edited lecture notes from a graduate course at the Computer Science department of Princeton University in Fall 2002. The course was my attempt to teach first year graduate students students many mathematical tools useful in theoretical computer science. Of course, the goal was too ambitious for a course with 12 three hour lectures. I had to relegate some topics to homework; these incl...

In linear regression, we need to avoid adding too much richness to the model. Therefore we need feature selection, or regularization to make our fitting curve smoother. Qualitatively, the original linear regression model is an optimization problem of the form min w m i=1 (w · x i − y i) 2 And the corresponding regularized version of the same problem is min w m i=1 (w · x i − y i) 2 + λw 2 2 , w...

We began the lecture with some final words on graphical models. Choosing a graphical model is akin to choosing a probability model for your data or choosing an algorithm. Each model has advantages and disadvantages that may make it more or less suitable for modeling your data. A graphical model is a representation of a probability model, so it also carries that model’s plusses and minuses. Grap...

In the last few lectures, we talked about various kinds of online learning problems. We started with the simplest case in which there is a perfect expert, who is always right. In the last lecture, we generalize this into the case in which we might not have a “perfect expert”, but instead we have a panel or subcommittee of experts, whose combined prediction is always right. We also introduced th...