Variational Inference on Deep Exponential Family by using Variational Inferences on Conjugate Models

In this paper, we propose a new variational inference method for deep exponentialfamily (DEF) models. Our method converts non-conjugate factors in a DEF model to easy-to-compute conjugate exponential-family messages. This enables local and modular updates similar to variational message passing, as well as stochastic natural-gradient updates similar to stochastic variational inference. Such updates make our algorithm highly suitable for large-scale learning. Our method exploits the structure of the deep network and can be useful to reduce the variance of stochastic methods for variational inference.