Stagewise Learning for Sparse Clustering of Discretely-Valued Data

We develop an algorithm to learn Bernoulli Mixture Models based on the principle that some variables are more informative than others. Working from an information-theoretic perspective, we propose both backward and forward schemes for selecting the informative ’active’ variables and using them to guide EM. The result is a stagewise EM algorithm, analogous to stagewise approaches to linear regression, that should be applicable to neuroscience (and other) datasets with confounding (or irrelevant) variables. Results on synthetic and MNIST datasets illustrate the approach.