\(\newcommand{\W}[1]{ \; #1 \; }\) \(\newcommand{\R}[1]{ {\rm #1} }\) \(\newcommand{\B}[1]{ {\bf #1} }\) \(\newcommand{\D}[2]{ \frac{\partial #1}{\partial #2} }\) \(\newcommand{\DD}[3]{ \frac{\partial^2 #1}{\partial #2 \partial #3} }\) \(\newcommand{\Dpow}[2]{ \frac{\partial^{#1}}{\partial {#2}^{#1}} }\) \(\newcommand{\dpow}[2]{ \frac{ {\rm d}^{#1}}{{\rm d}\, {#2}^{#1}} }\)

exp_reverse

View page source

Exponential Function Reverse Mode Theory

We use the reverse theory standard math function definition for the functions \(H\) and \(G\). The zero order forward mode formula for the exponential is

\[z^{(0)} = F ( x^{(0)} )\]

and for \(j > 0\),

\[z^{(j)} = x^{(j)} d^{(0)} + \frac{1}{j} \sum_{k=1}^{j} k x^{(k)} z^{(j-k)}\]

where

\[\begin{split}d^{(0)} = \left\{ \begin{array}{ll} 0 & \R{if} \; F(x) = \R{exp}(x) \\ 1 & \R{if} \; F(x) = \R{expm1}(x) \end{array} \right.\end{split}\]

For order \(j = 0, 1, \ldots\) we note that

\begin{eqnarray} \D{H}{ x^{(j)} } & = & \D{G}{ x^{(j)} } + \D{G}{ z^{(j)} } \D{ z^{(j)} }{ x^{(j)} } \\ & = & \D{G}{ x^{(j)} } + \D{G}{ z^{(j)} } ( d^{(0)} + z^{(0)} ) \end{eqnarray}

If \(j > 0\), then for \(k = 1 , \ldots , j\)

\begin{eqnarray} \D{H}{ x^{(k)} } & = & \D{G}{ x^{(k)} } + \D{G}{ z^{(j)} } \frac{1}{j} k z^{(j-k)} \\ \D{H}{ z^{(j-k)} } & = & \D{G}{ z^{(j-k)} } + \D{G}{ z^{(j)} } \frac{1}{j} k x^{(k)} \end{eqnarray}