We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by strongly improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant step...

This article considers the design and implementation of variable-timestep methods for simulating holonomically constrained mechanical systems. Symplectic variable stepsizes are briefly discussed, we then consider time-reparameterization techniques employing a time-reversible (symmetric) integration method to solve the equations of motion. We give several numerical examples, including a simulati...

This paper develops a Bregman operator splitting algorithm with variable stepsize (BOSVS) for solving problems of the form min{φ(Bu)+ 1/2‖Au− f ‖2}, where φ may be nonsmooth. The original Bregman Operator Splitting (BOS) algorithm employed a fixed stepsize, while BOSVS uses a line search to achieve better efficiency. These schemes are applicable to total variation (TV)-based image reconstructio...

The present paper describes implementation details of a new implementation of a BDF solver within MATLAB R . It is tuned to be most efficient when applied as a solver within FEMLAB R . The present code aims at a predictable and efficient performance for index-2 differential-algebraic equations arising within the method of lines. The main novelty is the consequent implementation of a predictive ...

We describe an extension of the Taylor method for numerical solution ODEs that uses Padé approximants to obtain extremely precise results. The accuracy results is essentially limited only by computer time and memory, provided one works in arbitrary precision. In this stepsize adjusted achieve desired (variable stepsize), while order expansion can be either fixed or changed at each iteration ord...

We propose a diagonal metric selection for variable metric proximal gradient method (VMPG). The proposed metric better captures the local geometry of the problem and provides improved convergence compared to the standard proximal gradient (PG) methods with Barzilai-Borwein (BB) stepsize selection. Further, we provide convergence guarantees for the proposed method and illustrate its advantages o...

Numerical solutions are proposed to fit the CanDecomp/ParaFac (CP) model of real three-way arrays, when the latter are both nonnegative and symmetric in two modes. In other words, a seminonnegative INDSCAL analysis is performed. The nonnegativity constraint is circumvented by means of changes of variable into squares, leading to an unconstrained problem. In addition, two globalization strategie...