Compressed Gaussian Process Manifold Regression

Nonparametric regression for massive numbers of samples (n) and features (p) is an important problem. We propose a Bayesian approach for scaling up Gaussian process (GP) regression to big n and p settings using random compression. The proposed compressed GP is particularly motivated by the setting in which features can be projected to a low-dimensional manifold with minimal loss of information about the response. Conditionally on a random compression matrix and a smoothness parameter, the posterior and posterior predictive distributions are available analytically. Running the analysis in parallel for many random compression matrices and smoothness parameters, model averaging is used to combine the results. The algorithm can be implemented rapidly even in very big n and p problems, has strong theoretical justification, and is found to yield state of the art predictive performance.