Abstract We study the foundations of variational inference, which frames posterior inference as an optimisation problem, for probabilistic programming. The dominant approach in practice is stochastic gradient descent. In particular, a variant using so-called reparameterisation estimator exhibits fast convergence traditional statistics setting. Unfortunately, discontinuities, are readily express...