Incremental Regression Function Construction with Small Landmarks

In automatic control and its related applications, many problems can be formulated as the regression estimation problem. In this paper, we construct a nonlinear regression model by using kernels as basis functions in a dictionary and applying the L1 norm as the regularizer. The regression function obtained from this model possesses the sparseness property where only a subset of points are used to represent the function. We call this subset of points as landmarks. It is a convex optimization problem. However, instead of using the standard optimization tools to solve a convex problem for a particular regularization value, we develop an efficient regularization path algorithm that can trace all solutions for all possible regularization parameter values. It overcomes the computational difficulty in model selection. Since the algorithm generally adds basis functions incrementally to improve the prediction accuracy, the regression function can be represented concisely with small landmarks.