Download Matrix Algebra

Matrix Algebra



Matrix algebra is a means of expressing large
numbers of calculations made upon ordered
sets of numbers.
Often referred to as Linear Algebra
Many equations would be completely
intractable if scalar mathematics had to be
used. It is also important to note that the
scalar algebra is under there somewhere.
Definitions - scalar

scalar - a number


denoted with regular type as is scalar
algebra
[1] or [a]
Definitions - vector

vector - a single row or column of numbers



denoted with bold small letters
row vector a =
1 2 3 4 5

column vector x =

 x1 
x 
 2
 x3 
 
 x4 
 x5 
Definitions - Matrix

A matrix is a set of rows and columns of
numbers
1 2 3
 4 5 6





Denoted with a bold Capital letter
All matrices (and vectors) have an order
- that is the number of rows x the
number of columns.
Thus A = 1 2 3
 4 5 6

 2 x 3 
Matrix Equality


Two matrices are equal iff (if and only
if) all of their elements are identical
Note: your data set is a matrix.
Matrix Operations




Addition and Subtraction
Multiplication
Transposition
Inversion
Addition and Subtraction


Two matrices may be added iff they are
the same order.
Simply add the corresponding elements
a11 a12  b11 b12  c11 c12 
a
  b
  c

a
b
c
 21 22   21 22   21 22 
a31 a32  b31 b32  c31 c32 
Addition and Subtraction
(cont.)

Where
a11  b11  c11
a12  b12  c12
a21  b21  c 21
a22  b22  c 22
a31  b31  c31
a32  b32  c32

Hence
1 2 4 6 5 8 
3 4  4 6  7 10

 
 

5 6 4 6 9 12
Scalar Multiplication

To multiply a scalar times a matrix,
simply multiply each element of the
matrix by the scalar quantity
a11 a12   2a11 2a12 
2



a
a
2
a
2
a
22 
 21 22   21
Matrix Multiplication (cont.)

To multiply a matrix times a matrix, we
write


A times B as AB
This is pre-multiplying
multiplying A by B.
B
by A, or post-
Matrix Multiplication (cont.)




In order to multiply matrices, they must
be conformable (the number of columns
in A must equal the number of rows in
B.)
an (mxn) x (nxp) = (mxp)
an (mxn) x (pxn) = cannot be done
a (1xn) x (nx1) = a scalar (1x1)
Matrix Multiplication (cont.)

The general rule for
matrix multiplication
is:
N
cij   aik bkj where i  1,2,..., M , and j  1,2,..., P
k 1
Matrix multiplication is not
Commutative


AB does not necessarily equal BA
(BA may even be an impossible
operation)
Yet matrix multiplication is
Associative

A(BC) = (AB)C
Special matrices

There are a number of special matrices





Square
Diagonal
Symmetric
Null
Identity
Square matrix

A square matrix is just what it sounds like, an nxn
matrix
 a11 a12
a
 21 a22
a31 a32

a41 a42

a13
a23
a33
a43
a14 

a24 
a34 

a44 
Square matrices are quite useful for describing the
properties or interrelationships among a set of
things – like the correlation matric for your dataset.
Diagonal Matrices

A diagonal matrix is a square matrix that
has values on the diagonal with all offdiagonal entities being zero.
0
0
a11 0
0 a

0
0
22


0
0 a33 0 


0
0 a44 
0
Symmetric Matrix


All of the elements in the upper right portion
of the matrix are identical to those in the
lower left.
For example, the correlation matrix
Identity Matrix

The identity matrix I is a diagonal
matrix where the diagonal elements all
equal one. It is used in a fashion
analogous to multiplying through by "1"
in scalar math.
1
0

0

0
0
1
0
0
0
0
1
0
0
0
0

1
Null Matrix

A square matrix where all elements equal
zero.
0
0

0

0

0
0
0
0
0
0
0
0
0
0
0

0
Not usually ‘used’ so much as sometimes the
result of a calculation.

Analogous to “a+b=0”
The Transpose of a Matrix A'



Taking the transpose is an operation
that creates a new matrix based on an
existing one.
The rows of A = the columns of A'
Hold upper left and lower right corners
and rotate 180 degrees.
Example of a transpose
1 4
1 2 3


A  2 5, A'  

4
5
6


3 6
The Transpose of a Matrix A'


If A = A', then A is symmetric (i.e. correlation
matrix)
If AA’ = A then A' is idempotent



The transpose of a sum = sum of transposes


(and A' = A)
Idempotent means applying the same operation
gives invariant results – e.g. 1 x 1 = 1 (see further…)
(A + B + C)’ = A’ + B’ + C’
The transpose of a product = the product of the
transposes in reverse order

(ABC)’ = C’B’A’
An example:


Suppose that you wish to obtain the
sum of squared errors from the vector
e. Simply pre-multiply e by its
transpose e'.
which, in matrices looks like

e' e  e1  e2  ..en
2
2
2

An example - cont

Since the matrix product is a scalar
found by summing the elements of the
vector squared.
The Determinant of a Matrix



The determinant of a matrix A is
denoted by |A|.
Determinants exist only for square
matrices.
They are a matrix characteristic, and
they are also difficult to compute
The Determinant for a 2x2 matrix

If A =

Then
a11 a12 
a

 21 a22 
A  a11a22  a12a21

That one is easy
The Determinant for a 3x3 matrix

If A =
a11 a12 a13 
a

a
a
22
23 
 21
a31 a32 a33 

Then
A = a11a22 a33 - a11a23a32 + a12 a23a31 - a12 a21a33 + a13a21a32 - a13a22 a31
Determinants


For 4 x 4 and up don't try. For those
interested, expansion by minors and
cofactors is the preferred method.
(However the spaghetti method works
well! Simply duplicate all but the last
column of the matrix next to the
original and sum the products of the
diagonals along the following pattern.)
Spaghetti Method of |A|
A = a11a22 a33 - a11a23a32 + a12 a23a31 - a12 a21a33 + a13a21a32 - a13a22 a31
a11 a12 a13  a11 a12 
a



a
a
a
a
21
22
23
21
22 


a31 a32 a33  a31 a32 
Spaghetti Method of |A|
A = a11a22 a33 - a11a23a32 + a12 a23a31 - a12 a21a33 + a13a21a32 - a13a22 a31
a11 a12 a13  a11 a12 
a



a
a
a
a
21
22
23
21
22 


a31 a32 a33  a31 a32 
Properties of Determinates

Determinants have several mathematical
properties which are useful in matrix
manipulations.





1 |A|=|A'|.
2. If a row of A = 0, then |A|= 0.
3. If every value in a row is multiplied by k,
then |A| = k|A|.
4. If two rows (or columns) are interchanged
the sign, but not value, of |A| changes.
5. If two rows are identical, |A| = 0.
Properties of Determinates



6. |A| remains unchanged if each
element of a row or each element
multiplied by a constant, is added to any
other row.
7. Det of product = product of Det's |AB|
= |A| |B|
8. Det of a diagonal matrix = product of
the diagonal elements
The Inverse of a Matrix (A-1)

For an nxn matrix A, there may be a B such
that AB = I = BA.




(The inverse is analogous to a reciprocal)
A matrix which has an inverse is nonsingular.
A matrix which does not have an inverse is
singular.
An inverse exists only if A  0
Inverse by Row or column
operations


Set up a tableau matrix
A tableau for inversions consists of the
matrix to be inverted post multiplied by
a conformable identity matrix.
Matrix Inversion by Tableau
Method

Rules:





You may interchange rows.
You may multiply a row by a scalar.
You may replace a row with the sum of that row
and another row multiplied by a scalar.
Every operation performed on A must be
performed on I
When you are done; A = I & I = A-1
The Tableau Method of Matrix
Inversion: An Example


Step 1: Set up Tableau
3  1 0 0
1 4
2 5
 0 1 0
4



1  3  2 0 0 1
Step 2: Add –2(Row 1)
to Row 2
3   1 0 0
1 4
0  3  2  2 1 0



1  3  2  0 0 1
Matrix Inversion – cont.
3   1 0 0
1 4
0  3  2  2 1 0



1  3  2  0 0 1


Step 3: Add –1(Row
1) to Row 3
3   1 0 0
1 4
0  3  2  2 1 0



0  7  5   1 0 1
Step 4: Multiply Row
2 by –1/3
3  1
0
0
1 4
0 1 2 / 3 2 / 3  1 / 3 0



0  7  5    1
0
1
Matrix Inversion – cont.
3  1
0
0
1 4
0 1 2 / 3 2 / 3  1 / 3 0



0  7  5    1
0
1


Step 5: Add –4 (Row 2) 1
0
to Row 1
0 1 / 3    5 / 3 4 / 3 0
  2 / 3  1 / 3 0
1
2
/
3



0  7  5    1
0
1
Step 6: Add 7(Row 2)
to Row 3
1 0 1 / 3    5 / 3 4 / 3 0
0 1 2 / 3   2 / 3


1
/
3
0



0 0  1 / 3  11 / 3  7 / 3 1
Matrix Inversion – cont.
1 0 1 / 3    5 / 3 4 / 3 0
0 1 2 / 3   2 / 3


1
/
3
0



0 0  1 / 3  11 / 3  7 / 3 1


0  2
 1 1
2 / 3   2 / 3  1 / 3 0
0 0  1 / 3 11 / 3  7 / 3 1
Step 7: Add Row 3 1 0
0 1
to Row 1

0  2
 1 1
1 0
 8

Step 9: Add 2(Row 0 1
0

5
2



3) to Row 2
0 0  1 / 3 11 / 3  7 / 3 1
Matrix Inversion – cont.


Step 9: Multiply Row 3
by -3
The original matrix is
now an identity, and
the original identity has
been transformed to the
Inverse
0  2
 1 1
1 0
0 1
 8

0

5
2



0 0  1 / 3 11 / 3  7 / 3 1
1 1 
1 0 0  2
0 1 0  8  5 2 



0 0 1  11 7  3
Checking the calculation

Remember AA-1=I
3  2
 1 1  1 0 0
1 4
2 5
  8  5 2   0 1 0
4


 

1  3  2  11 7  3 0 0 1

Thus
1 * 2  4 * 8  3 * 11  1
1 * 1  4 * 5  3 * 7   0
etc
Using Stata to do matrix Algebra

Try the following command set









matrix
matrix
matrix
matrix
matrix
matrix
matrix
matrix
matrix
input A= (1, 2 \ 3, 4)
list A
input B= (5, 6 \ 7, 8)
list B
define C = A + B
list C
X = (1, 4, 3 \ 2, 5, 4 \ 1, -3, -2)
Xinv = inv(X)
list Xinv
The Matrix Model

The multiple regression model may be
easily represented in matrix terms.
Y  XB  e

Where the Y, X, B and e are all
matrices of data, coefficients, or
residuals
The Matrix Model (cont.)

The matrices in Y  XB  e
represented by
 Y1 
Y 
2
Y 

 
 Yn 

 X 11 X 12
X X
21
22

X
 ... ...

 X n1 X n 2
... X ik 
... X 2 k 

... ... 

... X nk 
 B1 
B 
2
B  
 
 
 Bk 
are
 e1 
e 
2
e  
 
 
 en 
Note that we postmultiply X by B since this
order makes them conformable.
The Assumptions of the Model
Scalar Version







1. The ei's are normally distributed.
2. E(ei) = 0
3. E(ei2) = 2
4. E(eiej) = 0 (ij)
5. X's are nonstochastic with values fixed in repeated
samples and (Xik-Xbark)2/n is a finite nonzero
number.
6. The number of observations is greater than the
number of coefficients estimated.
7. No exact linear relationship exists between any of
the explanatory variables.
The Assumptions of the
Model: The Matrix Version

These same assumptions expressed in
matrix format are:



1. e  N(0,)
2.  = 2I
3. The elements of X are fixed in repeated
samples and (1/ n)X'X is nonsingular and
its elements are finite
Derivation of B's in matrix
notation




Given the matrix algebra model we can replicate
the least squares normal equations in matrix
format.
We need to minimize ee’ which is the sum of
squared errors
Setting the derivative equal to 0 we ultimately
get
B (X' X)1 X' Y
Note that X’X is called the sums-of-squares and
cross-products matrix.