The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from conjugate gradient in the stochastic (online) setting, using fast Hessian-gradient products to set up low-dimensional Krylov subspaces within individ...

The method of conjugate directions provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from conjugate gradient in the stochastic (online) setting, using fast Hessian-gradient products to set up low-dimensional Krylov subspaces within indivi...

Abstract: The existing literature predominantly concentrates on the utilization of the gradient descent algorithm for control systems’ design in power systems for stability enhancement. In this paper, various flavors of the Conjugate Gradient (CG) algorithm have been employed to design the online neuro-fuzzy linearization-based adaptive control strategy for Line Commutated Converters’ (LCC) Hig...

A new framework based on the curved Riemannian manifold is proposed to calculate the numerical solution of the Lyapunov matrix equation by using a natural gradient descent algorithm and taking the geodesic distance as the objective function. Moreover, a gradient descent algorithm based on the classical Euclidean distance is provided to compare with this natural gradient descent algorithm. Furth...

The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore a number of ways to adopt ideas from conjugate gradient in the stochastic setting, using fast Hessian-vector products to obtain curvature information cheaply. I...

Conjugate gradient methods are widely used for unconstrained optimization, especially large scale problems. Most of conjugate gradient methods don’t always generate a descent search direction, so the descent condition is usually assumed in the analyses and implementations. Dai and Yuan (1999) proposed the conjugate gradient method which generates a descent direction at every iteration. Yabe and...

We propose an Adaptive Stochastic Conjugate Gradient (ASCG) optimization algorithm for temporal medical image registration. This method combines the advantages of Conjugate Gradient (CG) method and Adaptive Stochastic Gradient Descent (ASGD) method. The main idea is that the search direction of ASGD is replaced by stochastic approximations of the conjugate gradient of the cost function. In addi...

Conjugate gradient methods are an important class of methods for solving unconstrained optimization problems, especially for large-scale problems. Recently, they have been studied in depth. In this paper, we further study the conjugate gradient method for unconstrained optimization. We focus our attention to the descent conjugate gradient method. This paper presents a modified conjugate gradien...

A nonlinear conjugate gradient algorithm which is a modification of the Dai and Yuan [Y.H. Dai and Y, Yuan, A nonlinear conjugate gradient method with a strong global convergence property, SIAM J. Optim., 10 (1999), pp.177-182.] conjugate gradient algorithm satisfying a parametrized sufficient descent condition with a parameter k δ is proposed. The parameter k δ is computed by means of the conj...