The standard conjugate gradient (CG) method uses orthogonality of the residues to simplify the formulas for the parameters necessary for convergence. In adaptive filtering, the sample-by-sample update of the correlation matrix and the cross-correlation vector causes a loss of the residue orthogonality in a modified online algorithm, which, in turn, results in loss of convergence and an increase...

The effective and efficient numerical solution of Riemann–Hilbert problems has been demonstrated in recent work. With the aid of ideas from the method of nonlinear steepest descent for Riemann– Hilbert problems, the resulting numerical methods have been shown numerically to retain accuracy as values of certain parameters become arbitrarily large. The primary aim of this paper is to prove that t...

A multi-layer neural network with multiple hidden layers was trained as an autoencoder using steepest descent, scaled conjugate gradient and alopex algorithms. These algorithms were used in different combinations with steepest descent and alopex used as pretraining algorithms followed by training using scaled conjugate gradient. All the algorithms were also used to train the autoencoders withou...

New methods make it possible to do approximate steepest descent in O(1) time per move for k-bounded pseudo-Boolean functions using stochastic local search. It is also possible to use the average fitness over the Hamming distance 2 neighborhood as a surrogate fitness function and still retain the O(1) time per move. These are average complexity results. In light of these new results, we examine ...

We study polynomials that are orthogonal with respect to a varying quartic weight exp(−N (x 2 /2+tx 4 /4)) for t < 0, where the orthogonal-ity takes place on certain contours in the complex plane. Inspired by developments in 2D quantum gravity, Fokas, Its, and Kitaev, showed that there exists a critical value for t around which the asymptotics of the recurrence coefficients are described in ter...

An algorithm is described to compute approximate classical trajectories as a boundary value problem with an integration step in the arc length. High-frequency motions are filtered out when a large integration step is used, maintaining the stability of the algorithm. At the limit of high filtering (large steps), but still offering an accurate description of the continuous path, the trajectory ap...

In this paper we explore different strategies to guide backpropagation algorithm used for training artificial neural networks. Two different variants of steepest descent-based backpropagation algorithm, and four different variants of conjugate gradient algorithm are tested. The variants differ whether or not the time component is used, and whether or not additional gradient information is utili...

Towards optimal Newton-type methods for nonconvex smooth optimization Coralia Cartis Coralia.Cartis (at) ed.ac.uk School of Mathematics, Edinburgh University We show that the steepest-descent and Newton methods for unconstrained non-convex optimization, under standard assumptions, may both require a number of iterations and function evaluations arbitrarily close to the steepest-descent’s global...

We address the problem of minimizing objectives from the class of piecewise differentiable functions whose nonsmoothness can be encapsulated in the absolute value function. They possess local piecewise linear approximations with a discrepancy that can be bounded by a quadratic proximal term. This overestimating local model is continuous but generally nonconvex. It can be generated in its abs-no...