Machine Learning Ensembles: An Empirical Study and Novel Approach

Two learning ensemble methods, Bagging and Boosting, have been applied to decision trees to improve classification accuracy over that of a single decision tree learner. We introduce Bagging and propose a variant of it — Improved Bagging — which, in general, outperforms the original bagging algorithm. We experiment on 22 datasets from the UCI repository, with emphasis on the ensemble’s accuracy as the main performance measure. The variant of Bagging that we propose utilizes the “out of bag” samples in determining a decision tree’s voting power, whereas in the classical Bagging algorithm, all trees have the same voting power. Our proposed algorithm creates bootstrap samples from the training data according to a forced probability distribution that emulates random sampling with replacement. We thus achieve a uniformly distributed training set for each base learner, and also identically-sized “out of bag” verification sets for each learner. Our 10-fold cross-validation results and single runs on large data sets show our novel Bagging variant to improve classification accuracy relative to both the original Bagging ensemble and standard Boosting.