We review three leading stochastic optimization methods—simulated annealing, genetic algorithms, and tabu search. In each case we analyze the method, give the exact algorithm, detail advantages and disadvantages, and summarize the literature on optimal values of the inputs. As a motivating example we describe the solution—using Bayesian decision theory, via maximization of expected utility—of a...

For supervised classification problems, it is well known that learnability is equivalent to uniform convergence of the empirical risks and thus to learnability by empirical minimization. Inspired by recent regret bounds for online convex optimization, we study stochastic convex optimization, and uncover a surprisingly different situation in the more general setting: although the stochastic conv...

This work presents the first projection-free algorithm to solve stochastic bi-level optimization problems, where objective function depends on solution of another problem. The proposed $\textbf{S}$tochastic $\textbf{Bi}$-level $\textbf{F}$rank-$\textbf{W}$olfe ($\textbf{SBFW}$) can be applied streaming settings and does not make use large batches or checkpoints. sample complexity SBFW is shown ...