BACKGROUND We address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. ...

Preliminary results obtained during the commissioning phase of the Soil Moisture and Ocean Salinity (SMOS) mission are described, devoting special attention to the characterization of the systematic errors found in the measurements and the corresponding impact in the retrieved salinity product. The identified issues and objectives to consolidate and improve the processing chain are also described.

Wollert’s (2002) paper on actuarial tests that are used for the prediction of sexual recidivism (ATSRs) showed that the recidivism rates for the score groups in the MnSOST-R developmental sample were inflated, and that they “shrank” by as much as 44 percentage points on cross-validation. Doren and Dow (in press) present an analysis of Wollert’s (2002) research and an alternative study that they...

Theoretical developments on cross validation (CV) have mainly focused on selecting one among a list of finite-dimensional models (e.g., subset or order selection in linear regression) or selecting a smoothing parameter (e.g., bandwidth for kernel smoothing). However, little is known about consistency of cross validation when applied to compare between parametric and nonparametric methods or wit...

The behaviour of model selection procedures based on diierent criteria such as cross-validation is investigated in a simulation study. Emphasis is on the relationship to the problem of estimating the prediction quality of a model.