The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore a number of ways to adopt ideas from conjugate gradient in the stochastic setting, using fast Hessian-vector products to obtain curvature information cheaply. I...

In this chapter, we investigate recently proposed nonlinear conjugate gradient (NCG) methods for shape optimization problems. We briefly introduce the as well corresponding theoretical background and their performance numerically. The obtained results confirm that NCG are efficient attractive solution algorithms

We propose and study kernel conjugate gradient methods (KCGM) with random projections for least-squares regression over a separable Hilbert space. Considering two types of generated by randomized sketches Nyström subsampling, we prove optimal statistical results respect to variants norms the algorithms under suitable stopping rule. Particularly, our show that if projection dimension is proporti...