Reverse Accumulation and Implicit Functions
چکیده
We begin by introducing a simple technique for using reverse accu mulation to obtain the rst derivatives of target functions which include in their construction the solution of systems of linear or nonlinear equations In the lin ear case solving Ay b for y corresponds to the adjoint operations b b v and A A yv where v is the solution to the adjoint equations vA y A more sophisticated construction applies in the nonlinear case We apply these techniques to obtain automatic numerical error estimates for calculated function values These error estimates include the e ects of inaccur ate equation solution as well as rounding error Our basic techniques can be generalized to functions which contain several linear or nonlinear implicit functions in their construction either serially or nested In the case of scalar valued target functions that include equation solu tion as part of their construction our algorithms involve at most the same order of computational e ort as the computation of the target function value regardless of the number of independent variables or the size of the systems of equations Introduction Automatic di erentiation is a set of techniques for obtaining derivatives of numerical functions to the same order of accuracy as the function values themselves but without the labour of forming explicit sym bolic expressions for the derivative functions Automatic di erentiation works by repeated use of the chain rule but applied to numerical expressions rather than to symbolic values The forward accumulation technique of automatic di erentiation associates with each program variable a vector containing partial derivatives of that vari able with respect to the independent variables The reverse accumulation tech nique builds a computational graph for the function evaluation with one graph node or Wengert variable corresponding to each successive value held by a program variable and associates with each node an adjoint variable containing the partial derivatives of the dependent variables with respect to that node The adjoint values are calculated numerically in the opposite order to the function evaluation whence the term reverse accumulation Reverse accumulation is of particular interest when the gradient or sensitivities of a single scalar valued target loss function is required as it allows the evaluation of the entire gradient vector of a scalar function at a computational cost of three function evaluations in total regardless of the number of independent variables For further details see and the references therein Many functions include in their computation the solution of a set of equations of the form y u where u R y R are column vectors of known and unknown quantities respectively and is a mapping R R R Provided the function is su ciently well behaved we can regard as the implicit de nition at u of a branch of the function y u de ned by y u u The eversion of this equation for a particular value of u so as to construct a corresponding value for y u is typically part of a larger computation of a dependent variable z f x y where the x are the independent variables y y u and the u are also functions of the x It is clearly desirable to extend the automatic di erentiation techniques so as to determine such quantities as the gradient row vector rx z The paper addresses these issues and proposes an approach to the automatic evaluation of rst derivatives in the context of equation solution by iterative xed point constructions More detailed analyses in this context were given by Griewank et al in the case where forward accumulation is used and by Christianson in the reverse case In this paper we examine a simple technique for using reverse accumulation to obtain rst derivatives of a target function dependent variable which in cludes implicit function eversions in its construction Some of the results and techniques presented here are generalizations of those given in others are completely new A corresponding analysis for the forward case has been given by Bartholomew Biggs In the next section following some mathematical preliminaries we show that the adjoint quantities u can also be de ned implicitly In section we use this to suggest an implementation approach which could be built on top of existing code In section we show how to obtain accurate error bounds for the e ects of inexact equation solution on target function values which are calculated via an implicit constructor In section we extend this to provide a uni ed analysis of the e ects of inexact solution of nonlinear equations and of rounding error and give an algorithm for calculating these error bounds numerically In section we discuss possible generalizations Adjoint Implicit Equations In what follows we assume a norm kbk for column vectors b and denote by the same symbol the corresponding operator norm kAk supfkAbk st kbk g We write kak for the norm of a row vector a considered as a linear operator on column vectors so that kak kakadj where T denotes the adjoint transpose Usually in practice we shall use the Euclidean norm in which case kak ka T k and the corresponding operator norm is the spectral norm but we do not assume this in what follows Because we are dealing with nite dimensional spaces all complete norms are equivalent and we can speak without ambiguity of an open neighbourhood of a point However the equivalence bounds can be arbitrarily large We regard the gradient operator r as an adjoint operator mapping scalars into row vectors and column vectors into Jacobian matrices If is a mapping of the form y u R R R then we write y u to denote ry ru respectively when these derivatives exist At any such point y u y u y y u u y u is a map R q R L R R R On a region where is everywhere de ned we say that is Lipschitz with respect to a given norm if the map y u y u is Lipschitz with respect to the induced operator norm The results we give below remain true in the case where is uniformly continuous rather than Lipschitz The methods of proof are the same but the construction of bounds is notationally more cumbersome De nition Fix the norm k k on R R this induces norms on R R f g R f g R and on L R R R Choose u R p y R q and let U Y be bounded open convex subsets of R R respectively such that u U y Y Since we are interested in local behaviour with respect to u we lose nothing by restricting our attention to domains of this form We say that Y U R is a well behaved implicit mapping if the fol lowing four conditions are satis ed a y u b is continuously di erentiable throughout Y U c is Lipschitz with constant C throughout Y U for some constant C d y is uniformly invertible throughout Y U ie there is some constant such that k y y u k for all u in U and y in Y Note that under these conditions there will always exist a constant with k u y u k for all u in U and y in Y We could relax condition d in De nition as the following Lemma shows Lemma Suppose that Y U R satis es conditions a b and c in De nition and that y is invertible at y u Then there exist convex sets Y U with y Y Y and u U U such that restricted to Y U is a well behaved implicit mapping Proof Choose Y and U small enough to ensure that ky y k and ku u k are both less than C k y y u k for all y Y u U De ne A y y u A y y u D A A A