\(\newcommand{\W}[1]{ \; #1 \; }\) \(\newcommand{\R}[1]{ {\rm #1} }\) \(\newcommand{\B}[1]{ {\bf #1} }\) \(\newcommand{\D}[2]{ \frac{\partial #1}{\partial #2} }\) \(\newcommand{\DD}[3]{ \frac{\partial^2 #1}{\partial #2 \partial #3} }\) \(\newcommand{\Dpow}[2]{ \frac{\partial^{#1}}{\partial {#2}^{#1}} }\) \(\newcommand{\dpow}[2]{ \frac{ {\rm d}^{#1}}{{\rm d}\, {#2}^{#1}} }\)

forward_theory

View page source

The Theory of Forward Mode

Taylor Notation

In Taylor notation, each variable corresponds to a function of a single argument which we denote by t (see Section 10.2 of Evaluating Derivatives ). Here and below \(X(t)\), \(Y(t)\), and Z ( t ) are scalar valued functions and the corresponding p-th order Taylor coefficients row vectors are \(x\), \(y\) and \(z\); i.e.,

\begin{eqnarray} X(t) & = & x^{(0)} + x^{(1)} * t + \cdots + x^{(p)} * t^p + o( t^p ) \\ Y(t) & = & y^{(0)} + y^{(1)} * t + \cdots + y^{(p)} * t^p + o( t^p ) \\ Z(t) & = & z^{(0)} + z^{(1)} * t + \cdots + z^{(p)} * t^p + o( t^p ) \end{eqnarray}

For the purposes of this section, we are given \(x\) and \(y\) and need to determine \(z\).

Binary Operators

Addition

\begin{eqnarray} Z(t) & = & X(t) + Y(t) \\ \sum_{j=0}^p z^{(j)} * t^j & = & \sum_{j=0}^p x^{(j)} * t^j + \sum_{j=0}^p y^{(j)} * t^j + o( t^p ) \\ z^{(j)} & = & x^{(j)} + y^{(j)} \end{eqnarray}

Subtraction

\begin{eqnarray} Z(t) & = & X(t) - Y(t) \\ \sum_{j=0}^p z^{(j)} * t^j & = & \sum_{j=0}^p x^{(j)} * t^j - \sum_{j=0}^p y^{(j)} * t^j + o( t^p ) \\ z^{(j)} & = & x^{(j)} - y^{(j)} \end{eqnarray}

Multiplication

\begin{eqnarray} Z(t) & = & X(t) * Y(t) \\ \sum_{j=0}^p z^{(j)} * t^j & = & \left( \sum_{j=0}^p x^{(j)} * t^j \right) * \left( \sum_{j=0}^p y^{(j)} * t^j \right) + o( t^p ) \\ z^{(j)} & = & \sum_{k=0}^j x^{(j-k)} * y^{(k)} \end{eqnarray}

Division

\begin{eqnarray} Z(t) & = & X(t) / Y(t) \\ x & = & z * y \\ \sum_{j=0}^p x^{(j)} * t^j & = & \left( \sum_{j=0}^p z^{(j)} * t^j \right) * \left( \sum_{j=0}^p y^{(j)} * t^j \right) + o( t^p ) \\ x^{(j)} & = & \sum_{k=0}^j z^{(j-k)} y^{(k)} \\ z^{(j)} & = & \frac{1}{y^{(0)}} \left( x^{(j)} - \sum_{k=1}^j z^{(j-k)} y^{(k)} \right) \end{eqnarray}

Standard Math Functions

Suppose that \(F\) is a standard math function and

\[Z(t) = F[ X(t) ]\]

Differential Equation

All of the standard math functions satisfy a differential equation of the form

\[B(u) * F^{(1)} (u) - A(u) * F (u) = D(u)\]

We use \(a\), \(b\) and \(d\) to denote the p-th order Taylor coefficient row vectors for \(A [ X (t) ]\), \(B [ X (t) ]\) and \(D [ X (t) ]\) respectively. We assume that these coefficients are known functions of \(x\), the p-th order Taylor coefficients for \(X(t)\).

Taylor Coefficients Recursion Formula

Our problem here is to express \(z\), the p-th order Taylor coefficient row vector for \(Z(t)\), in terms of these other known coefficients. It follows from the formulas above that

\begin{eqnarray} Z^{(1)} (t) & = & F^{(1)} [ X(t) ] * X^{(1)} (t) \\ B[ X(t) ] * Z^{(1)} (t) & = & \{ D[ X(t) ] + A[ X(t) ] * Z(t) \} * X^{(1)} (t) \\ B[ X(t) ] * Z^{(1)} (t) & = & E(t) * X^{(1)} (t) \end{eqnarray}

where we define

\[E(t) = D[X(t)] + A[X(t)] * Z(t)\]

We can compute the value of \(z^{(0)}\) using the formula

\[z^{(0)} = F ( x^{(0)} )\]

Suppose by induction (on \(j\)) that we are given the Taylor coefficients of \(E(t)\) up to order \(j-1\); i.e., \(e^{(k)}\) for \(k = 0 , \ldots , j-1\) and the coefficients \(z^{(k)}\) for \(k = 0 , \ldots , j\). We can compute \(e^{(j)}\) using the formula

\[e^{(j)} = d^{(j)} + \sum_{k=0}^j a^{(j-k)} * z^{(k)}\]

We need to complete the induction by finding formulas for \(z^{(j+1)}\). It follows from the definition of \(E(t)\) that

\[\left( \sum_{k=0}^j b^{(k)} * t^k \right) * \left( \sum_{k=1}^{j+1} k z^{(k)} * t^{k-1} \right) = \left( \sum_{k=0}^j e^{(k)} * t^k \right) * \left( \sum_{k=1}^{j+1} k x^{(k)} * t^{k-1} \right) + o( t^p )\]

Setting the left and right side coefficients of \(t^j\) equal, and using the formula for Multiplication , we obtain

\begin{eqnarray} \sum_{k=0}^j b^{(k)} (j+1-k) z^{(j+1-k)} & = & \sum_{k=0}^j e^{(k)} (j+1-k) x^{(j+1-k)} \\ z^{(j+1)} & = & \frac{1}{j+1} \frac{1}{ b^{(0)} } \left( \sum_{k=0}^j e^{(k)} (j+1-k) x^{(j+1-k)} - \sum_{k=1}^j b^{(k)} (j+1-k) z^{(j+1-k)} \right) \\ z^{(j+1)} & = & \frac{1}{j+1} \frac{1}{ b^{(0)} } \left( \sum_{k=1}^{j+1} k x^{(k)} e^{(j+1-k)} - \sum_{k=1}^j k z^{(k)} b^{(j+1-k)} \right) \end{eqnarray}

This completes the induction that computes \(e^{(j)}\) and \(z^{(j+1)}\).

Cases that Apply Recursion Above

exp_forward	Exponential Function Forward Mode Theory
log_forward	Logarithm Function Forward Mode Theory
sqrt_forward	Square Root Function Forward Mode Theory
sin_cos_forward	Trigonometric and Hyperbolic Sine and Cosine Forward Theory
atan_forward	Inverse Tangent and Hyperbolic Tangent Forward Mode Theory
asin_forward	Inverse Sine and Hyperbolic Sine Forward Mode Theory
acos_forward	Inverse Cosine and Hyperbolic Cosine Forward Mode Theory
pow_forward	Power Function Forward Mode Theory

Special Cases

tan_forward	Tangent and Hyperbolic Tangent Forward Taylor Polynomial Theory
erf_forward	Error Function Forward Taylor Polynomial Theory