This tutorial describes the classic method of conjugate directions: the generalization of the conjugate-gradient method in iterative least-square inversion. I derive the algebraic equations of the conjugate-direction method from general optimization principles. The derivation explains the “magic” properties of conjugate gradients. It also justifies the use of conjugate directions in cases when ...

The Stokes problem plays an important role in computational fluid dynamics since it is encountered in the time discretization of (incompressible) Navier-Stokes equations by operator-splitting methods [2, 3]. Space discretization of the Stokes problem leads to large scale ill-conditioned systems. The Uzawa (preconditioned) conjugate gradient method is an efficient method for solving the Stokes p...

A global convergence theorem for unconstrained minimization algorithms with an eecient line search is derived. The theorem applies to a new version of the conjugate gradient method derived here in terms of minimizing the eeect of zigzagging. The global convergence condition makes much weaker demands on the line search than previous methods. Local Q-linear convergence in a neighborhood of a stro...

In this paper, we propose a couple of new Stochastic Strictly Contractive PeacemanRachford Splitting Method (SCPRSM), called Stochastic SCPRSM (SS-PRSM) and Stochastic Conjugate Gradient SCPRSM (SCG-PRSM) for large-scale optimization problems. The two types of Stochastic PRSM algorithms respectively incorporate stochastic variance reduced gradient (SVRG) and conjugate gradient method. Stochasti...

The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore a number of ways to adopt ideas from conjugate gradient in the stochastic setting, using fast Hessian-vector products to obtain curvature information cheaply. I...

The Conjugate Gradient Method is the most prominent iterative method for solving sparse systems of linear equations. Unfortunately, many textbook treatments of the topic are written with neither illustrations nor intuition, and their victims can be found to this day babbling senselessly in the corners of dusty libraries. For this reason, a deep, geometric understanding of the method has been re...

In our papers [Inverse Problems, 15, 309-327,1999] and [Numer. Math., 88, 347-365, 2001] we proposed algorithm REGINN being an inexact Newton iteration for the stable solution of nonlinear ill-posed problems. REGINN consists of two components: the outer iteration, which is a Newton iteration stopped by the discrepancy principle, and an inner iteration, which computes the Newton correction by so...

For a large-scale quadratic programming problem with separable objective function, a variant of the conjugate gradient method can effectively be applied to the dual problem. In this paper, we consider a block-parallel modification of the conjugate gradient method, which is suitable for implementation on a parallel computer. More precisely, the method proceeds in a block Jacobi manner and execut...

This paper explores the convergence of nonlinear conjugate gradient methods without restarts, and with practical line searches. The analysis covers two classes of methods that are globally convergent on smooth, nonconvex functions. Some properties of the Fletcher-Reeves method play an important role in the first family, whereas the second family shares an important property with the Polak-Ribir...

For solving systems of linear algebraic equations with blockfivediagonal matrices arising in geoelectrics and diffusion problems, the parallel matrix square root method, conjugate gradient method with preconditioner, conjugate gradient method with regularization, and parallel matrix sweep algorithm are proposed and some of them are implemented numerically on multi-core CPU Intel. Investigation ...