Spectral properties of the kernel matrix and their relation to kernel methods in machine learning

This chapter serves as a brief introduction to the supervised learning setting and kernel methods. Moreover, several results from linear algebra, probability theory, and functional analysis are reviewed which will be used throughout the thesis. 2.1 Some notational conventions We begin by introducing some basic notational conventions. The sets N, Z, R, C denote the natural, integer, real, and complex numbers. Vectors will be denoted by lowercase letters, whereas matrices will be denoted by bold uppercase letters. Random variables will be denoted by uppercase letters. The individual entries of vectors and matrices are denoted by square brackets. For example, x ∈ R is a vector with coefficients [x]i. The matrix A has entries [A]ij . Vector and matrix transpose is denoted by x>. Sometimes, the set of square n × n matrices are denoted by Mn, and the set of general n × m matrices by Mn,m. The set of eigenvalues of a square matrix A is denoted by λ(A). For a symmetric n × n matrixA, we will always assume that the eigenvalues and eigenvectors are sorted in non-increasing order with eigenvalues repeated according to their multiplicity. The eigenvalues of A are thus λ1(A) ≥ . . . ≥ λn(A). We use the following standard norms on finite-dimensional vector spaces. Let x ∈ R and A ∈Mn. Then, ‖x‖ = √√√√ n ∑ i=1 [x]i , ‖A‖ = max x : ‖x‖6=0 ‖Ax‖ ‖x‖ . (2.1) A useful upper bound on ‖A‖ is given by ‖A‖ ≤ n max 1≤i,j≤n |[A]ij |. (2.2) Another matrix norm we will encounter is the Frobenius norm ‖A‖F = √√√√ n ∑