We propose a generic method for iteratively approximating various second-order gradient steps - Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient - in linear time per iteration, using special curvature matrix-vector products that can be computed in O(n). Two recent acceleration techniques for on-line learning, matrix momentum and stochastic meta-descent (SMD), implement this appro...

Bootstrap testing of nonlinear models normally requires at least one nonlinear estimation for every bootstrap sample. We show how to reduce computational costs by performing only a fixed, small number of Newton or quasi-Newton steps for each bootstrap sample. The number of steps is smaller for likelihood ratio tests than for other types of classical tests, and smaller for Newton’s method than f...

We present a phase-based potential ordering that is an extension of the Cascade ordering introduced by Appleyard and Cheshire [1]. The ordering is valid for both two-phase and three-phase flow, and it can handle countercurrent flow due to gravity and/or capillarity. We show how this ordering can be used to reduce the nonlinear algebraic system that arise from the fully-implicit method (FIM) int...

Many state-of-the-art approaches to trajectory optimization and optimal control are intimately related to standard Newton methods. For researchers that work in the intersections of machine learning, robotics, control, and optimization, such relations are highly relevant but sometimes hard to see across disciplines, due also to the different notations and conventions used in the disciplines. The...

We study the problem of learning a kernel matrix from an apriori kernel and training data. An unconstrained convex optimization formulation is proposed, with an arbitrary convex smooth loss function on kernel entries and a LogDet divergence for regularization. Since the number of variables is of order O(n), standard Newton and quasi-Newton methods are too time-consuming. An operator form Hessia...

Aims. We describe in detail the nature of XMM-Newton EPIC background and its various complex components, summarising the new findings of the XMM-Newton EPIC background working group, and provide XMM-Newton background blank sky event files for use in the data analysis of diffuse and extended sources. Methods. Blank sky event file data sets are produced from the stacking of data, taken from 189 o...

Classical iteration methods for linear systems, such as Jacobi iteration, can be accelerated considerably by Krylov subspace methods like GMRES. In this paper, we describe how inexact Newton methods for nonlinear problems can be accelerated in a similar way and how this leads to a general framework that includes many well-known techniques for solving linear and nonlinear systems, as well as new...

Signaling layer protocols specified within the Next Steps in Signaling (NSIS) framework may rely on the General Internet Signaling Transport (GIST) protocol to handle authorization. Still, the signaling layer protocol above GIST itself may require separate authorization to be performed when a node receives a request for a certain kind of service or resources. This document presents a generic mo...