The SGD-QN algorithm is a stochastic gradient descent algorithm that makes careful use of secondorder information and splits the parameter update into independently scheduled components. Thanks to this design, SGD-QN iterates nearly as fast as a first-order stochastic gradient descent but requires less iterations to achieve the same accuracy. This algorithm won the “Wild Track” of the first PAS...

I Simple proof of linear convergence. I For convex functions, equivalent to several of the above conditions. I For non-convex functions, weakest assumption while still guaranteeing global minimizer. ? We generalize the PL condition to analyze proximal-gradient methods. ? We give simple new analyses in a variety of settings: I Least-squares and logistic regression. I Randomized coordinate descen...

Poor (even random) starting points for learning/training/optimization are common in machine learning. In many settings, the method of Robbins and Monro (online stochastic gradient descent) is known to be optimal for good starting points, but may not be optimal for poor starting points — indeed, for poor starting points Nesterov acceleration can help during the initial iterations, even though Ne...

Despite their popularity, the practical performance of asynchronous stochastic gradient descent methods (ASGD) for solving large scale machine learning problems are not as good as theoretical results indicate. We adopt and analyze a synchronous K-step averaging stochastic gradient descent algorithm which we call K-AVG. We establish the convergence results of KAVG for nonconvex objectives, and s...

This paper provides a tutorial introduction to the constant modulus (CM) criterion for blind fractionally spaced equalizer (FSE) design via a (stochastic) gradient descent algorithm such as the constant modulus algorithm (CMA). The topical divisions utilized in this tutorial can be used to help catalog the emerging literature on the CM criterion and on the behavior of (stochastic) gradient desc...

We train two convolutional neural networks on the POFA and NimStim datasets to identify individuals and identify emotions, respectively. In order to train these neural networks, we use two separate optimization procedures, the minFunc package and stochastic gradient descent. The minFunc optimization package achieved a 95.8% on the training set and achieved a 90.0% accuracy on the partitioned te...

We propose a Bayesian approach for recursively estimating the classifier weights in online learning of a classifier ensemble. In contrast with past methods, such as stochastic gradient descent or online boosting, our approach estimates the weights by recursively updating its posterior distribution. For a specified class of loss functions, we show that it is possible to formulate a suitably defi...

‎based on an eigenvalue analysis‎, ‎a new proof for the sufficient‎ ‎descent property of the modified polak-ribière-polyak conjugate‎ ‎gradient method proposed by yu et al‎. ‎is presented‎.

Two types of low cost-per-iteration gradient descent methods have been extensively studied in parallel. One is online or stochastic gradient descent ( OGD/SGD), and the other is randomzied coordinate descent (RBCD). In this paper, we combine the two types of methods together and propose online randomized block coordinate descent (ORBCD). At each iteration, ORBCD only computes the partial gradie...