Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key components: i) adaptive preconditioners (as in ADAgrad or RMSprop), and ii) ada...

We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently. More specifically, the proposed algorithm only needs Õ( −10/3) stochastic gradient evaluations to converge to an approximate local minimum x, which satisfies...

In this paper, we examine a class of non-convex stochastic optimization problems which we call variationally coherent, and which properly includes pseudo-/quasiconvex and star-convex optimization problems. To solve such problems, we focus on the widely used stochastic mirror descent (SMD) family of algorithms (which contains stochastic gradient descent as a special case), and we show that the l...

Shape optimization of the fine scale geometry of elastic objects is investigated under stochastic loading. Thus, the object geometry is described via parametrized geometric details placed on a regular lattice. Here, in a two dimensional set up we focus on ellipsoidal holes as the fine scale geometric details described by the semiaxes and their orientation. Optimization of a deterministic cost f...

Stochastic optimization, also known as optimization under uncertainty, is studied by over a dozen communities, often (but not always) with different notational systems and styles, typically motivated by different problem classes (or sometimes different research questions) which often lead to different algorithmic strategies. This resulting “jungle of stochastic optimization” has produced a high...

The mean completion time of a stochastic process may be rendered finite and minimised by a judiciously chosen restart protocol, which may either be stochastic or deterministic. Here we study analytically an arbitrary stochastic search subject to an arbitrary restart protocol, each characterised by a distribution of waiting times. By a direct enumeration of paths we construct the joint distribut...

In wireless sensor networks (WSNs), there generally exist many different objective functions to be optimized. In this paper, we propose a stochastic multiobjective optimization approach to solve such kind of problem. We first formulate a general multiobjective optimization problem. We then decompose the optimization formulation through Lagrange dual decomposition and adopt the stochastic quasig...

Risk averse stochastic optimization is investigated in the context of elastic shape optimization, allowing for microstructures in the admissible shapes. In particular, a two-stage model for shape optimization under stochastic loading with risk averse cost functionals is combined with a two-scale approach for the simulation of microstructured materials. The microstructure is composed of an elast...

In this paper we focus on the application of global stochastic optimization methods to extremum estimators. We propose a general stochastic method—the master method —which includes several stochastic optimization algorithms as a particular case. The proposed method is sufficiently general to include the Solis–Wets method, the improving hit-and-run algorithm, and a stochastic version of the zigz...