We revisit an optimization strategy recently introduced by the authors to compute numerical approximations of minimizers for optimal control problems governed by scalar conservation laws in the presence of shocks. We focus on the one-dimensional (1-D) Burgers equation. This new descent strategy, called the alternating descent method, in the inviscid case, distinguishes and alternates descent di...

In solving a linear system with iterative methods, one is usually confronted with the dilemma of having to choose between cheap, inefficient iterates over sparse search directions (e.g., coordinate descent), or expensive iterates in well-chosen search directions (e.g., conjugate gradients). In this paper, we propose to interpolate between these two extremes, and show how to perform cheap iterat...

| An algorithm is proposed for training the single-layered per-ceptron. The algorithm follows successive steepest descent directions with respect to the perceptron cost function, taking care not to increase the number of misclassiied patterns. The problem of nding these directions is stated as a quadratic programming task, to which a fast and eeective solution is proposed. The resulting algorit...

We address the solution of convex-constrained nonlinear systems of equations where the Jacobian matrix is unavailable or its computation/storage is burdensome. In order to efficiently solve such problems, we propose a new class of algorithms which are “derivativefree” both in the computation of the search direction and in the selection of the steplength. Search directions comprise the residuals...

We study the convergence of two stochastic approximation algorithms with randomized directions: the simultaneous perturbation stochastic approximation algorithm and the random direction Kiefer–Wolfowitz algorithm. We establish deterministic necessary and sufficient conditions on the random directions and noise sequences for both algorithms, and these conditions demonstrate the effect of the “ra...

We study the convergence of two stochastic approximation algorithms with randomized directions: the simultaneous perturbation stochastic approximation algorithm and the random direction Kiefer–Wolfowitz algorithm. We establish deterministic necessary and sufficient conditions on the random directions and noise sequences for both algorithms, and these conditions demonstrate the effect of the “ra...

We analyze the variance of stochastic gradients along negative curvature directions in certain nonconvex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show that contrary to the case of isotropic noise this variance is proportional to the magnitude of the corresponding eigenvalues and not decreasing in the dimensiona...

An algorithm is proposed for training the single-layered perceptron. The algorithm follows successive steepest descent directions with respect to the perceptron cost function, taking care not to increase the number of misclassified patterns. The problem of finding these directions is stated as a quadratic programming task, to which a fast and effective solution is proposed. The resulting algori...

It has been shown recently that the efficiency of direct search methods that use opportunistic polling in positive spanning directions can be improved significantly by reordering the poll directions according to descent indicators built from simplex gradients. The purpose of this paper is twofold. First, we analyze the properties of simplex gradients of nonsmooth functions in the context of dir...