A Unified Bias-Variance Decomposition and its Applications

This paper presents a unified bias-variance decomposition that is applicable to squared loss, zero-one loss, variable misclassification costs, and other loss functions. The unified decomposition sheds light on a number of significant issues: the relation between some of the previously-proposed decompositions for zero-one loss and the original one for squared loss, the relation between bias, variance and Schapire et al.’s (1997) notion of margin, and the nature of the trade-off between bias and variance in classification. While the biasvariance behavior of zero-one loss and variable misclassification costs is quite different from that of squared loss, this difference derives directly from the different definitions of loss. We have applied the proposed decomposition to decision tree learning, instancebased learning and boosting on a large suite of benchmark data sets, and made several significant observations.