Model Monitor User’s Guide version 1.0

Model Monitor (M) is a software tool for studying the effect of distribution shifts the performance of machine learning algorithms. Most practical work in data mining assumes that data distributions are stationary, which is to say they do not change over time. As such, classifiers tend to be most effective when the data on which they are evaluated (test data) is similar to the data on which the model was built (training data). In real life, however, shifts in distribution do occur, and a model that is robust to changes in distribution will be significantly more valuable in the long run than one which is not. With this fact in mind, we designed Model Monitor around two basic tasks. First, it can be used to determine whether a distribution shift has occurred. Given separate train and test datasets (or a single dataset with a train-test split), Model Monitor will test each individual attribute of the dataset for distribution shift, using methods from [2], and report any features that have changed. Second, Model Monitor can evaluate the robustness of a particular classifier to certain types of distribution shift. Using any commandline classifier, or a classifier from the WEKA machine learning environment, Model Monitor will run the classifier and report its performance under any one of a number of chosen performance measures or loss measures. In this way, researchers can better design classifiers that are resistant to distribution shifts, or practitioners can choose a model that performs well under an anticipated set of future conditions.