Four decades after their invention, quasiNewton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newton methods can be interpreted as approximations of Bayesian linear regression u...

Analyses of the convergence properties of general quasi-Newton methods are presented, particular attention being paid to how the approximate solutions and the iteration matrices approach their final values. It is further shown that when Broyden's algorithm is applied to linear systems, the error norms are majorised by a superlinearly convergent sequence of an unusual kind.

Training in the random neural network (RNN) is generally speci®ed as the minimization of an appropriate error function with respect to the parameters of the network (weights corresponding to positive and negative connections). We propose here a technique for error minimization that is based on the use of quasi-Newton optimization techniques. Such techniques o€er more sophisticated exploitation ...

We first recall some properties of infinite tridiagonal matrices considered as matrix transformations in sequence spaces of the forms sξ , sξ , s (c) ξ , or lp(ξ). Then, we give some results on the finite section method for approximating a solution of an infinite linear system. Finally, using a quasi-Newton method, we construct a sequence that converges fast to a solution of an infinite linear ...

A classical model of Newton iterations which takes into account some error terms is given by the quasi-Newton method, which assumes perturbed Jacobians at each step. Its high convergence orders were characterized by Dennis and Moré [Math. Comp. 28 (1974), 549–560]. The inexact Newton method constitutes another such model, since it assumes that at each step the linear systems are only approximat...

In this paper, we present a method for solving the nite nonlinear min-max problem. By using quasi-Newton methods, we approximately solve a sequence of diierentiable subproblems where, for each subproblem, the cost function to minimize is a global regularization underestimating the nite maximum function. We show that every cluster point of the sequence generated is a stationary point of the min-...

This paper develops and analyzes a generalization of the Broyden class of quasiNewton methods to the problem of minimizing a smooth objective function f on a Riemannian manifold. A condition on vector transport and retraction that guarantees convergence and facilitates efficient computation is derived. Experimental evidence is presented demonstrating the value of the extension to the Riemannian...

This paper is dedicated to Claude Lemaréchal on the occasion of his 65th birthday. We take this opportunity to thank him deeply for the great moments we have had discussing with him (not only about math). His vision and his ability to put ideas into words has helped us deepen our understanding of optimization. This work builds on one of his lines of research: using convex analysis and nonlinear...

In this paper, we present a quasi-Newton (QN) algorithm for joint independent subspace analysis (JISA). JISA is a recently proposed generalization of independent vector analysis (IVA). JISA extends classical blind source separation (BSS) to jointly resolve several BSS problems by exploiting statistical dependence between latent sources across mixtures, as well as relaxing the assumption of stat...

We introduce a general criterion for blindly extracting a subset of sources in instantaneous mixtures. We derive the corresponding estimation equations and generalize them based on arbitrary nonlinear separating functions. A quasi-Newton algorithm for minimizing the criterion is presented, which reduces to the FastICA algorithm in the case when only one source is extracted. The asymptotic distr...