atomic_four_mat_mul

View page source

Atomic Matrix Multiply Class: Example Implementation

Syntax

atomic_mat_mul mat_mul ( name )
call_id = mat_mul . set ( n_left , n_middle , n_right )
mat_mul . get ( call_id , n_left , n_middle , n_right )
mat_mul ( call_id , x , y )

Purpose

Construct an atomic operation that computes the matrix product C = A * B .

n_left

This is the row dimension of the matrices A and C . This is an argument (return value) for the set (get ) routine.

n_middle

This is the column dimension of the matrix A and row dimension of the matrix B This is an argument (return value) for the set (get ) routine.

n_right

This is the column dimension of the matrices B and C . This is an argument (return value) for the set (get ) routine.

call_id

This is a return value (argument) for the set (get ) routine.

x

We use x to denote the argument to the atomic function. The size of this vector must be

n = n_left * n_middle + n_middle * n_right

The matrix A is stored in row major order at the beginning of x ; i.e. its ( i , k ) element is

A ( i , k ) = x [ i * n_middle + k ]

The matrix B is stored in row major order at the end of x ; i.e. its ( k , j ) element is

B ( k , j ) = x [ n_left * n_middle + k * n_right + j ]

y

We use y to denote the result of the atomic function. The size of this vector must be m = n_middle * n_right . The matrix C is stored in row major order in y ; i.e. its ( i , k ) element is

C ( i , j ) = y [ i * n_right + j ]

Theory

Forward

For \(k = 0 , \ldots\), the k-th order Taylor coefficient \(C^{(k)}\) is given by

\[C^{(k)} = \sum_{\ell = 0}^{k} A^{(\ell)} B^{(k-\ell)}\]

Matrix Argument Scalar Valued Function

Suppose \(\bar{F}\) is the derivative of the scalar value function \(s(F)\) with respect to the matrix \(F\); i.e.,

\[\bar{F}_{i,j} = \frac{ \partial s } { \partial F_{i,j} }\]

Also suppose that \(t\) is a scalar valued argument and

\[F(t) = D(t) E(t)\]

It follows that

\[F'(t) = D'(t) E(t) + D(t) E'(t)\]
\[(s \circ F)'(t) = \R{tr} [ \bar{F}^\R{T} F'(t) ]\]
\[= \R{tr} [ \bar{F}^\R{T} D'(t) E(t) ] + \R{tr} [ \bar{F}^\R{T} D(t) E'(t) ]\]
\[= \R{tr} [ E(t) \bar{F}^\R{T} D'(t) ] + \R{tr} [ \bar{F}^\R{T} D(t) E'(t) ]\]

Letting \(E(t) = 0\) and \(D(t) = \Delta^{i,j} (t)\) (where \(\Delta^{i,j} (t)\) is the matrix that is zero, except for \(i = j\) where it is \(t\)) we have

\[\bar{D}_{i,j} = \frac{ \partial s } { \partial D_{i,j} } = (s \circ F)'(t) = \R{tr} [ E(t) \bar{F}^\R{T} \Delta^{i,j}(1) ]\]
\[\bar{D}_{i,j} = \sum_k D_{j,k} \bar{F}^\R{T}_{k,i} = \sum_k \bar{F}_{i,k} E^\R{T}_{k,j}\]
\[\bar{D} = \bar{F} E^\R{T}\]

Letting \(D(t) = 0\) and \(E(t) = \Delta^{i,j} (t)\) we have

\[\bar{E}_{i,j} = \frac{ \partial s } { \partial E_{i,j} } = (s \circ F)'(t) = \R{tr} [ \bar{F}^\R{T} D(t) \Delta^{i,j} ]\]
\[\bar{E}_{i,j} = \sum_k \bar{F}^\R{T}_{j,k} C_{k,i} = \sum_k D^\R{T}_{i,k} \bar{F}_{k,j}\]
\[\bar{E} = D^\R{T} \bar{F}\]

Reverse

Reverse mode eliminates \(C^{(k)}\) as follows: for \(\ell = 0, \ldots , k\),

\[\bar{A}^{(\ell)} = \bar{A}^{(\ell)} + \bar{C}^{(k)} [ B^{(k-\ell)} ] ^\R{T}\]
\[\bar{B}^{(k-\ell)} = \bar{B}^{(k-\ell)} + [ A^{(\ell)} ]^\R{T} \bar{C}^{(k)}\]

Contents

Name

Title

atomic_four_mat_mul_implement

Implementing Atomic Matrix Multiply

atomic_four_mat_mul_forward.cpp

Atomic Matrix Multiply Forward Mode: Example and Test

atomic_four_mat_mul_reverse.cpp

Atomic Matrix Multiply Reverse Mode: Example and Test

atomic_four_mat_mul_sparsity.cpp

Atomic Matrix Multiply Sparsity Patterns: Example and Test

atomic_four_mat_mul_rev_depend.cpp

Atomic Matrix Multiply Reverse Dependency: Example and Test

atomic_four_mat_mul_identical_zero.cpp

Atomic Matrix Multiply Identical Zero: Example and Test