Fast hierarchical Gaussian processes

While the framework of Gaussian process priors for functions is very flexible and has a number of advantages, its use within a fully Bayesian hierarchical modeling framework has been limited due to computational constraints. Most often, simple models are fit, with hyperparameters learned by maximum likelihood. But this approach understates the posterior uncertainty in inference. We consider priors over kernel hyperparameters, thus inducing a very flexible Bayesian hierarchical modeling framework in which we perform inference using MCMC not just for the posterior function but also for the posterior over the kernel hyperparameters. We address the central challenge of computational efficiency with MCMC by exploiting separable structure in the covariance matrix corresponding to the kernel, yielding significant gains in time and memory efficiency. Our method can be conveniently implemented in a probabilistic programming language (Stan), is widely applicable to any setting involving structured kernels, and immediately enables a number of useful features, including kernel learning through novel prior specifications, learning nonparametric priors over categorical variables, clustering through a factor analysis approach, and missing observations. We demonstrate our methods on real and synthetic spatiotemporal datasets.