Research Proposal: Instance-Based Weighting Scheme in Ensemble Methods

Recent research in classification problems has mostly concentrated on ensemble methods that construct a set of base classifiers instead of a single classifier. An unlabeled instance is then classified by taking a vote of the base classifiers’ predictions of its class label. Ensemble methods like Bagging and AdaBoost have been shown to outperform the individual base classifiers when the base inducer that produces the base classifiers is unstable. An inducer is said to be unstable if a slight change in the training examples results in a very different classifier being constructed. Further, the idea of ensemble methods also gives rise to the use of error correcting output code technique in enhancing accuracy in multi-class classification problems. In ensemble methods, the vote of each base classifier either receives the same weight (e.g. Bagging and Arcing) or is weighted according to the estimated error rate (e.g. AdaBoost). Based on the recent work of the PI, we propose an instance-based approach of assigning weights to the base classifiers that may enhance the performance of ensemble methods. Instead of assigning a weight to a base classifier that is fixed for all instances, we attempt to assign a weight that is based upon how well we think the base classifier is going to predict on the label of the test instance. Intuitively, given an unlabeled instance x, if there is some indication that the best classifier’s prediction is correct then we should increase its weight. Otherwise, we should reduce its weight. In our preliminary investigation (see Section C.3), on nine UC Irvine datasets where AdaBoost did not perform well, with accuracy between 70% to 80%, we manage to improve the prediction accuracy on all datasets with an average improvement of more than 1%. This suggests that such instance-based weighting scheme does enhance the accuracy of ensemble methods. The implementation is currently still quite crude. We believe it can be further fine-tuned to achieve an even higher accuracy. Section C.3 proposes some ideas that will be explored in this project.