A Replicator Dynamics Approach to Collective Feature Engineering in Random Forests

It has been demonstrated how random subspaces can be used to create a Diversified Random Forest, which in turn can lead to better performance in terms of predictive accuracy. Motivated by the fact that each subsforest is built using a set of features that can overlap with those sets of features in other subforests, we hypothesise that using Replicator Dynamics can perform a collective feature engineering, by allowing subforests with better performance to grow and those with lower performance to shrink. In this paper, we propose a new method to further improve the performance of Diversified Random Forest using Replicator Dynamics which has been used extensively in evolutionary game dynamics. A thorough experimental study on 15 real datasets showed favourable results, demonstrating the potential of the proposed method. Some experiments reported a boost in predictive accuracy of over 10% consistently, evidencing the effectiveness of the iterative feature engineering achieved through the Replicator Dynamics procedure.