An efficient model identification algorithm for a large class of linear-in-the-parameters models is introduced that simultaneously optimises the model approximation ability, sparsity and robustness. The derived model parameters in each forward regression step are initially estimated via the orthogonal least squares (OLS), followed by being tuned with a new gradient-descent learning algorithm ba...

This paper proposes a new way of using forward selection of explanatory variables in regression or canonical redundancy analysis. The classical forward selection method presents two problems: a highly inflated Type I error and an overestimation of the amount of explained variance. Correcting these problems will greatly improve the performance of this very useful method in ecological modeling. T...

Traditionally, feature selection methods work directly on labeled examples. However, the availability of labeled examples cannot be taken for granted for many real world applications, such as medical diagnosis, forensic science, fraud detection, etc, where labeled examples are hard to find. This practical problem calls the need for “semi-supervised feature selection” to choose the optimal set o...

This paper defines and studies a variable selection procedure called Testing-Based Forward Model Selection. The procedure inductively selects covariates which increase predictive accuracy into a working statistical regression model until a stopping criterion is met. The stopping criteria and selection criteria are defined using statistical hypothesis tests. The paper explicitly describes a test...

Recently, considerable interest has focused on variable selection methods in regression situations where the number of predictors, p, is large relative to the number of observations, n. Two commonly applied variable selection approaches are the Lasso, which computes highly shrunk regression coefficients, and Forward Selection, which uses no shrinkage. We propose a new approach, “Forward-Lasso A...

The problem of feature selection for Support Vector Machines (SVMs) classification is investigated in the linear two classes case. We suggest a new method of feature selection based on ranking scores derived from SVMs. We analyze the retraining effects on the ranking rules based on these scores. Our features selection algorithm consists in a forward selection strategy according to the decreasin...

In this report we summarize the recent paper [Taylor et al., 2014] which proposes new inference tools for methods that perform variable selection and estimation in an adaptive regression. Although this paper mainly studies forward stepwise regression (FS) and least angle regression (LAR), the approach in this paper is not limited to these cases. This paper describes how to carry out exact infer...

An alternative solution to the model structure selection problem is introduced by conducting a forward search through the many possible candidate model terms initially and then performing an exhaustive all subset model selection on the resulting model. An example is included to demonstrate that this approach leads to dynamically valid nonlinear models.