Cos 511: Theoretical Machine Learning 2 Relative Entropy and Chernoff Bounds 2.1 Relative Entropy

In the previous lecture, we saw that PAC learning is not enough to model real learning problems. We may not desire or even may not be able to find a hypothesis that is consistent with the training data. So we introduced a more general model in which the data is generated as a pair (x, y) from an unknown distribution D. And we defined the generalization error to be errD(h) = Pr(x,y)∼D[h(x) 6= y] and also the empirical error of the training set x1, ..., xm to be the fraction of mistakes on the training set . ˆ err(h) = 1 m |{i : h(xi) 6= yi|. We also showed that in order to get an appropriate bound for the generalization error, it is enough to show that |err(h) − ˆ err(h)| ≤ . And today, we will introduce some powerful tools to find the desired bounds.