----------------------------------------------------
lines 6-213 of file: xrst/theory/reverse_theory.xrst
----------------------------------------------------

{xrst_begin reverse_theory}

The Theory of Reverse Mode
##########################

Taylor Notation
***************
In Taylor notation, each variable corresponds to
a function of a single argument which we denote by *t*
(see Section 10.2 of
:ref:`Bib@Evaluating Derivatives` ).
Here and below
:math:`X(t)`, :math:`Y(t)`, and *Z* ( *t* ) are scalar valued functions
and the corresponding *p*-th order Taylor coefficients row vectors are
:math:`x`, :math:`y` and :math:`z`; i.e.,

.. math::
   :nowrap:

   \begin{eqnarray}
   X(t) & = & x^{(0)} + x^{(1)} * t + \cdots + x^{(p)} * t^p + O( t^{p+1} ) \\
   Y(t) & = & y^{(0)} + y^{(1)} * t + \cdots + y^{(p)} * t^p + O( t^{p+1} ) \\
   Z(t) & = & z^{(0)} + z^{(1)} * t + \cdots + z^{(p)} * t^p + O( t^{p+1} )
   \end{eqnarray}

For the purposes of this discussion,
we are given the *p*-th order Taylor coefficient row vectors
:math:`x`, :math:`y`, and  :math:`z`.
In addition, we are given the partial derivatives of a scalar valued function

.. math::

   G ( z^{(j)} , \ldots , z^{(0)}, x, y)

We need to compute the partial derivatives of the scalar valued function

.. math::

   H ( z^{(j-1)} , \ldots , z^{(0)}, x, y)  =
   G ( z^{(j)}, z^{(j-1)} , \ldots , z^{(0)}, x , y )

where :math:`z^{(j)}` is expressed as a function of the
*j-1*-th order Taylor coefficient row
vector for :math:`Z` and the vectors :math:`x`, :math:`y`; i.e.,
:math:`z^{(j)}` above is a shorthand for

.. math::

   z^{(j)} ( z^{(j-1)} , \ldots , z^{(0)}, x, y )

If we do not provide a formula for
a partial derivative of :math:`H`, then that partial derivative
has the same value as for the function :math:`G`.

Binary Operators
****************

Addition
========
The forward mode formula for
:ref:`forward_theory@Binary Operators@Addition` is

.. math::

   z^{(j)} =  x^{(j)} + y^{(j)}

If follows that for :math:`k = 0 , \ldots , j`
and :math:`l = 0 , \ldots , j-1`

.. math::
   :nowrap:

   \begin{eqnarray}
   \D{H}{ x^{(k)} } & = &
   \D{G}{ x^{(k)} }  + \D{G}{ z^{(k)} } \\
   \\
   \D{H}{ y^{(k)} } & = &
   \D{G}{ y^{(k)} }  + \D{G}{ z^{(k)} }
   \\
   \D{H}{ z^{(l)} } & = & \D{G}{ z^{(l)} }
   \end{eqnarray}

Subtraction
===========
The forward mode formula for
:ref:`forward_theory@Binary Operators@Subtraction` is

.. math::

   z^{(j)} =  x^{(j)} - y^{(j)}

If follows that for :math:`k = 0 , \ldots , j`

.. math::
   :nowrap:

   \begin{eqnarray}
   \D{H}{ x^{(k)} } & = &
   \D{G}{ x^{(k)} }  - \D{G}{ z^{(k)} } \\
   \\
   \D{H}{ y^{(k)} } & = &
   \D{G}{ y^{(k)} }  - \D{G}{ z^{(k)} }
   \end{eqnarray}

Multiplication
==============
The forward mode formula for
:ref:`forward_theory@Binary Operators@Multiplication` is

.. math::

   z^{(j)} = \sum_{k=0}^j x^{(j-k)} * y^{(k)}

If follows that for :math:`k = 0 , \ldots , j`
and :math:`l = 0 , \ldots , j-1`

.. math::
   :nowrap:

   \begin{eqnarray}
   \D{H}{ x^{(j-k)} } & = &
   \D{G}{ x^{(j-k)} }  +
   \sum_{k=0}^j \D{G}{ z^{(j)} } y^{(k)}
   \\
   \D{H}{ y^{(k)} } & = &
   \D{G}{ y^{(k)} }  +
   \sum_{k=0}^j \D{G}{ z^{(j)} } x^{(j-k)}
   \end{eqnarray}

Division
========
The forward mode formula for
:ref:`forward_theory@Binary Operators@Division` is

.. math::

   z^{(j)} =
   \frac{1}{y^{(0)}}
   \left(
      x^{(j)} - \sum_{k=1}^j z^{(j-k)} y^{(k)}
   \right)

If follows that for :math:`k = 1 , \ldots , j`

.. math::
   :nowrap:

   \begin{eqnarray}
   \D{H}{ x^{(j)} } & = &
   \D{G}{ x^{(j)} }  + \D{G}{ z^{(j)} } \frac{1}{y^{(0)}}
   \\
   \D{H}{ z^{(j-k)} } & = &
   \D{G}{ z^{(j-k)} }  - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} y^{(k)}
   \\
   \D{H}{ y^{(k)} } & = &
   \D{G}{ y^{(k)} }  - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} z^{(j-k)}
   \\
   \D{H}{ y^{(0)} } & = &
   \D{G}{ y^{(0)} }  - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} \frac{1}{y^{(0)}}
   \left(
      x^{(j)} - \sum_{k=1}^j z^{(j-k)} y^{(k)}
   \right)
   \\
   & = &
   \D{G}{ y^{(0)} }  - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} z^{(j)}
   \end{eqnarray}

Standard Math Functions
***********************
The standard math functions have only one argument.
Hence we are given the partial derivatives of a scalar valued function

.. math::

   G ( z^{(j)} , \ldots , z^{(0)}, x)

We need to compute the partial derivatives of the scalar valued function

.. math::

   H ( z^{(j-1)} , \ldots , z^{(0)}, x)  =
   G ( z^{(j)}, z^{(j-1)} , \ldots , z^{(0)}, x)

where :math:`z^{(j)}` is expressed as a function of the
*j-1*-th order Taylor coefficient row
vector for :math:`Z` and the vector :math:`x`; i.e.,
:math:`z^{(j)}` above is a shorthand for

.. math::

   z^{(j)} ( z^{(j-1)} , \ldots , z^{(0)}, x )

Contents
********
{xrst_toc_table
   xrst/theory/exp_reverse.xrst
   xrst/theory/log_reverse.xrst
   xrst/theory/sqrt_reverse.xrst
   xrst/theory/sin_cos_reverse.xrst
   xrst/theory/atan_reverse.xrst
   xrst/theory/asin_reverse.xrst
   xrst/theory/acos_reverse.xrst
   xrst/theory/tan_reverse.xrst
   xrst/theory/erf_reverse.xrst
   xrst/theory/pow_reverse.xrst
}

{xrst_end reverse_theory}