This work studies constrained stochastic optimization problems where the objective and constraint functions are convex expressed as compositions of functions. The problem arises in context fair classification, regression, design queuing systems. Of particular interest is large-scale setting an oracle provides gradients constituent functions, goal to solve with a minimal number calls oracle. Owi...

Stein variational gradient descent (SVGD) and its variants have shown promising successes in approximate inference for complex distributions. In practice, we notice that the kernel used SVGD-based methods has a decisive effect on empirical performance. Radial basis function (RBF) with median heuristics is common choice previous approaches, but unfortunately this proven to be sub-optimal. Inspir...

We consider a decentralized learning setting in which data is distributed over nodes graph. The goal to learn global model on the without involving any central entity that needs be trusted. While gossip-based stochastic gradient descent (SGD) can used achieve this objective, it incurs high communication and computation costs, since has wait for all local models at converge. To speed up converge...

We use differential equations based approaches to provide some physics insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton’s methods as well as their Nesterov’s accelerated variants in a unified framework motivated by...

The method of conjugate directions provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from conjugate gradient in the stochastic (online) setting, using fast Hessian-gradient products to set up low-dimensional Krylov subspaces within indivi...

Fluid mixing and turbulent processes such as double diffusion are chaotic by nature can be very difficult to parameterize. Experts have called for further investigation into parameterizing other vertical the implication that it may on large-scale ocean climate models. Interference from lateral flows often make field-data-driven parameterizations isolated experiments much more accurate results. ...