Stochastic Gradient Descent on Riemannian Manifolds

Averaging Stochastic Gradient Descent on Riemannian Manifolds

We consider the minimization of a function defined on a Riemannian manifold M accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on M to an averaged iterate sequence with a robust and fast O(1/n) convergence rate. We then present an application of our...

A Gradient-Descent Method for Curve Fitting on Riemannian Manifolds

Given data points p0, . . . , pN on a closed submanifold M of R n and time instants 0 = t0 < t1 < . . . < tN = 1, we consider the problem of finding a curve γ on M that best approximates the data points at the given instants while being as “regular” as possible. Specifically, γ is expressed as the curve that minimizes the weighted sum of a sum-of-squares term penalizing the lack of fitting to t...

Stochastic variational principle on riemannian manifolds

Variational Stochastic Gradient Descent

In Bayesian approach to probabilistic modeling of data we select a model for probabilities of data that depends on a continuous vector of parameters. For a given data set Bayesian theorem gives a probability distribution of the model parameters. Then the inference of outcomes and probabilities of new data could be found by averaging over the parameter distribution of the model, which is an intr...

Byzantine Stochastic Gradient Descent

This paper studies the problem of distributed stochastic optimization in an adversarial setting where, out of the m machines which allegedly compute stochastic gradients every iteration, an α-fraction are Byzantine, and can behave arbitrarily and adversarially. Our main result is a variant of stochastic gradient descent (SGD) which finds ε-approximate minimizers of convex functions in T = Õ ( 1...