Download Linear Algebra and Matrices

Linear Algebra and Matrices
Methods for Dummies
20th October, 2010
Melaine Boly
Christian Lambert
Overview
•
•
•
•
•
Definitions-Scalars, vectors and matrices
Vector and matrix calculations
Identity, inverse matrices & determinants
Eigenvectors & dot products
Relevance to SPM and terminology
Linear Algebra & Matrices, MfD 2010
Part I
Matrix Basics
Linear Algebra & Matrices, MfD 2010
Scalar
• A quantity (variable), described by a single
real number
e.g. Intensity of each
voxel in an MRI scan
Linear Algebra & Matrices, MfD 2010
Vector
• Not a physics vector (magnitude, direction)
EXAMPLE:
 x1 
 x 2
VECTOR=

i.e. A column 
of numbers 

xn
 
Linear Algebra & Matrices, MfD 2010
Matrices
• Rectangular display of vectors in rows and columns
• Can inform about the same vector intensity at different
times or different voxels at the same time
• Vector is just a n x 1 matrix
 x11 x12 x13 
 x 21 x 22 x 23


 x31 x32 x33
Linear Algebra & Matrices, MfD 2010
Matrices
Matrix locations/size defined as rows x columns (R x C)
 d11
D  d 21
d 31
1 4 7 
A  2 5 8
3 6 9
Square (3 x 3)
d12
d 22
d 32
d13 
d 23 
d 33 
d i j : ith row, jth column
 x11 x12 x13 
x11 x12 x13
 x21
 
x
22
x
23
x11 x12  x13
 
  x21
x
22
x
23
11 xx33
12  x13
 
  x21
 x31
xx32
x
22
x
23
11 xx33
12  x13
 
  x21
 x31
xx32
x
22
x
23
  x 21
 
 x31
x32 x 22
x33x 23

 x31
x32 x33 
 x31 x32 x33
3 dimensional (3 x 3 x 5)
Linear Algebra & Matrices, MfD 2010
1 4 
A  2 5
3 6
Rectangular (3 x 2)
Matrices in MATLAB
Description
Matrix(X)
Type into MATLAB
X=[1 4 7;2 5 8;3 6 9]
Meaning
1 4 7 
X  2 5 8
3 6 9
;=end of a row
Reference matrix values (X(row,column))
Note the : refers to all of row or column and , is the divider between rows and columns
3rd row
X(3, :)
2nd Element of 3rd column
X(2,3)
Elements 2&3 of column 2
X( [1 2], 2)
Special types of matrix
All zeros size 3x1
zeros(3,1)
All ones size 2x2
ones(2,2)
Linear Algebra & Matrices, MfD 2010
3
6 9
8
 4
 
5
0
 
0
0
 
1

1
1

1
Transposition
1 
b  1 
2
column
b  1 1 2
T
row
1 2 3
A  5 4 1
6 7 4
Linear Algebra & Matrices, MfD 2010
d  3 4 9
row
1 5 6 
AT  2 4 7
3 1 4
 3


T
d   4
9
column
Matrix Calculations
Addition
– Commutative: A+B=B+A
– Associative: (A+B)+C=A+(B+C)
2
AB  
2
Subtraction
- By adding a negative matrix
Linear Algebra & Matrices, MfD 2010
4 1 0  2  1 4  0 3 4






5 3 1 2  3 5  1  5 6
Scalar multiplication
Scalar * matrix = scalar multiplication
Linear Algebra & Matrices, MfD 2010
Matrix Multiplication
“When A is a mxn matrix & B is a kxl matrix, AB is only possible
if n=k. The result will be an mxl matrix”
Simply put, can ONLY perform A*B IF:
Number of columns in A = Number of rows in B
m
n
l
A1 A2 A3
B13 B14
A4 A5 A6
x
A7 A8 A9
B15 B16
k
= m x l matrix
B17 B18
A10 A11 A12
Hint:
1) If you see this message in MATLAB:
??? Error using ==> mtimes
Inner matrix dimensions must agree
Linear Algebra & Matrices, MfD 2010
-Then columns in A is not equal to rows in B
Matrix multiplication
• Multiplication method:
Sum over product of respective rows and columns
Matlab does all this for you!
Simply type: C = A * B
Hints:
1) You can work out the size of the output (2x2). In MATLAB, if you pre-allocate a
matrix this size (e.g. C=zeros(2,2)) then the calculation is quicker
Linear Algebra & Matrices, MfD 2010
Matrix multiplication
• Matrix multiplication is NOT commutative
i.e the order matters!
– AB≠BA
• Matrix multiplication IS associative
– A(BC)=(AB)C
• Matrix multiplication IS distributive
– A(B+C)=AB+AC
– (A+B)C=AC+BC
Linear Algebra & Matrices, MfD 2010
Identity matrix
A special matrix which plays a similar role as the number 1 in number
multiplication?
For any nxn matrix A, we have A In = In A = A
For any nxm matrix A, we have In A = A, and A Im = A (so 2 possible matrices)
If the answers always A, why use an identity matrix?
Can’t divide matrices, therefore to solve may problems have to use the inverse. The
identity is important in these types of calculations.
Linear Algebra & Matrices, MfD 2010
Identity matrix
Worked
example
A I3 = A
for a 3x3 matrix:
1
2
3
4
5
6
7
8
9
X
1
0
0
0
1
0
0
0
1
=
1+0+0
0+2+0
0+0+3
4+0+0
0+5+0
0+0+6
7+0+0
0+8+0
0+0+9
• In Matlab: eye(r, c) produces an r x c identity matrix
Linear Algebra & Matrices, MfD 2010
Part II
More Advanced Matrix Techniques
Linear Algebra & Matrices, MfD 2010
Vector components
& orthonormal base
• A given vector (a b) can be summarized by its
components, but only in a particular base (set of
axes; the vector itself can be independent from the
choice of this particular base).
y axis

v
example

a and b are the components of v
in the given base (axes chosen
for expression of the coordinates
in vector space)
Orthonormal base: set of vectors chosen to express the components
of the others, perpendicular to each other and all with norm (length) = 1
Linear Algebra & Matrices, MfD 2010
b

v
a
x axis
Linear combination &
dimensionality
Vectorial space: space defined by different vectors (for example for
dimensions…).
The vectorial space defined by some vectors is a space that contains them and all
the vectors that can be obtained by multiplying these vectors by a real number
then adding them (linear combination).
A matrix A (mn) can itself be decomposed in as many vectors as its number of
columns (or lines). When decomposed, one can represent each column of the
matrix by a vector. The ensemble of n vector-column defines a vectorial space
proper to matrix A.
Similarly, A can be viewed as a matricial representation of this ensemble of
vectors, expressing their components in a given base.
Linear Algebra & Matrices, MfD 2010
Linear dependency and rank
If one can find a linear relationship between the lines or columns of a matrix,
then the rank of the matrix (number of dimensions of its vectorial space) will not
be equal to its number of column/lines – the matrix will be said to be rankdeficient.
Example

2
X1
24
When representing the vectors, we see that
x1 and x2 are superimposed. If we look
better, we see that we can express one by a
linear combination of the other: x2 = 2 x1.
The rank of the matrix will be 1.
In parallel, the vectorial space defined will
has only one dimension.
y

x 2
4
x1  12
4

x2
2
2
x1

x
1
Linear Algebra & Matrices, MfD 2010
2
Linear dependency and rank
• The rank of a matrix corresponds to the dimensionality of the vectorial space
defined by this matrix. It corresponds to the number of vectors defined by the
matrix that are linearly independents from each other.
• Linealy independent vectors are vectors defining each one one more
dimension in space, compared to the space defined by the other vectors. They
cannot be expressed by a linear combination of the others.
Note. Linearly independent vectors are not
necessarily orthogonal (perpendicular).
Example: take 3 linearly independent vectors
x1, x2 et x3.
Vectors x1 and x2 define a plane (x,y)
And vector x3 has an additional non-zero
component in the z axis.
But x3 is not perpendicular to x1 or x2.
Linear Algebra & Matrices, MfD 2010

x3

x1

x2
plan
Eigenvalues et eigenvectors
One can represent the vectors from matrix X (eigenvectors of A) as a set of orthogonal vectors
(perpendicular), and thus representing the different dimensions of the original matrix A.
The amplitude of the matrix A in these different dimensions will be given by the eigenvalues
corresponding to the different eigenvectors of A (the vectors composing X).
Note: if a matrix is rank-deficient, at least one of its eigenvalues is zero.
For A’:
u1, u2 = eigenvectors
k1, k2 = eigenvalues
In Principal Component Analysis (PCA), the matrix is decomposed into eigenvectors and
eigenvalues AND the matrix is rotated to a new coordinate system such that the greatest variance
by any projection of the data comes to lie on the first coordinate (called the first principal
component), the second greatest variance on the second coordinate, and so on.
Linear Algebra & Matrices, MfD 2010
Vector Products
Two vectors:
 x1 

x
x
2
 

 x3 

 y1 

y
y
2
 

 y3 

Inner product XTY is a scalar
(1xn) (nx1)
Inner product = scalar
xT y  x1
x2
x3 
 y1 
3
y   x y  x y  x y 
xi yi

1 1
2 2
3 3
 2
i 1
 y3 
Outer product = matrix
x1 
 
xy T  x 2 y1

x 3 

y2
x1y1

y 3   x 2 y1

x 3 y1
Linear Algebra & Matrices, MfD 2010
x1y 2
x2y2
x3y2
x1y 3  Outer product XYT is a matrix

x 2 y 3 
(nx1) (1xn)
x 3 y 3 

Scalar product of vectors
Calculate the scalar product of two vectors is equivqlent to
make the projection of one vector on the other one. One can
indeed show that x1 x2 = x1 . x2 . cos where  is the
angle that separates two vectors when they have both the
same origin.

x1

cos

x1 x2 =   .   . cos

x1 x2

x2
In parallel, if two vectors are orthogonal, their scalar product is
zero: the projection of one onto the other will be zero.
Linear Algebra & Matrices, MfD 2010
Determinants
Le déterminant d’une matrice est un nombre scalaire
représentant certaines propriétés intrinsèques de cette
matrice. Il est noté detA ou |A|.
Sa définition est un détour indispensable avant d’aborder
l’opération correspondant à la division de matrices, avec
le calcul de l’inverse.
Linear Algebra & Matrices, MfD 2010
For a matrix 11:
Determinants
det(a11)a11
For a matrix 22:
a11a12 a a a a
a21a22 11 22 12 21
For a matrix 33:
a11 a12 a13
a21 a22 a23 = a11a22a33+a12a23a31+a13a21a32–a11a23a32 –a12a21a33 –a13a22a31
a31 a32 a33
= a11(a22a33 –a23a32)–a12(a21a33 –a23a31)+a13(a21a32 –a22a31)
The determinant of a matrix can be calculate by multiplying each element of one
of its lines by the determinant of a sub-matrix formed by the elements that
stay when one suppress the line and column containing this element. One
give to the obtained product the sign (-1)i+j.
Linear Algebra & Matrices, MfD 2010
Determinants
•Determinants can only be found for square matrices.
•For a 2x2 matrix A, det(A) = ad-bc. Lets have at closer look at that:
det(A) =
[ ]
a
c
b
d
= ad - bc
• In Matlab: det(A) = det(A)
The determinant gives an idea of the ’volume’ occupied by the matrix in vector
space
A matrix A has an inverse matrix A-1 if and only if det(A)≠0.
Linear Algebra & Matrices, MfD 2010
Determinants
The determinant of a matrix is zero if and only if there exist a linear relationship
between the lines or the columns of the matrix – if the matrix is rank-deficient.
In parallel, one can define the rank of a matrix A as the size of the largest
square sub-matrix of A that has a non-zero determionant.

2
X1
24
Here x1 and x2 are superimposed in space,
because one can be expressed by a linear
combination of the other: x2 = 2 x1.
y

x 2
4
x1  12
4

x2
The determinant of the matrix X will thus be
zero.
2
2
The largest square sub-matrix with a nonzero determinant will be a matrix of 1x1 =>
the rank of the matrix is 1.
x1

x
1
Linear Algebra & Matrices, MfD 2010
2
Determinants
• In a vectorial space of n dimensions, there will be
no more than n linearly independent vectors.
• If 3 vectors (21) x’1, x’2, x’3 are represented by a matrix X’:
Graphically, we have:
Here x3 can be expressed by
a linear combination of x1 and
x2.
The determinant of the matrix
X’ will thus be zero.
The largest square sub-matrix
with a non-zero determinant
will be a matrix of 2x2 => the
rank of the matrix is 2.
y
y
3
3
(1,2)
2

x2
x1

Linear Algebra & Matrices, MfD 2010
  
x2 ax1 bx3
(2,2)
2
(3,1)

x3
1
1
 
X' 1223
21
2
3
1
x

ax1

x2

bx3
1
2
3
x
Determinants
The notions of determinant, of the rank of a matrix and of linear dependency are
closely linked.
Take a set of vectors x1, x2,…,xn, all with the same number of elements: these
vectors are linearly dependent if one can find a set of scalars c1, c2,…,cn non
equal to zero such as:
c1 x1+ c2 x2+…+ cn xn= 0
A set of vectors are linearly dependent if one of then can be expressed as a
linear combination of the others. They define in space a smaller number of
dimensions than the total number of vectors in the set. The resulting matrix will
be rank-deficient and the determinant will be zero.
Similarly, if all the elements of a line or column are zero, the determinant of the
matrix will be zero.
If a matrix present two rows or columns that are equal, its determinant will also
be zero
Linear Algebra & Matrices, MfD 2010
Matrix inverse
• Definition. A matrix A is called nonsingular or invertible if there
exists a matrix B such that:
1
1
-1
2
X
2
3
-1
3
1
3
1
3
=
2+1
3 3
-1 + 1
3 3
-2+ 2
3 3
1+2
3 3
=
1
0
0
1
• Notation. A common notation for the
inverse of a matrix A is A-1. So:
• The inverse matrix is unique when it exists. So if A is invertible, then
A-1 is also invertible and then (AT)-1 = (A-1)T
• In Matlab: A-1 = inv(A)
Linear Algebra & Matrices, MfD 2010
•Matrix division: A/B= A*B-1
Matrix inverse
• For a XxX square matrix:
• The inverse matrix is:
• E.g.: 2x2 matrix
For a matrix to be invertible, its determinant has to be non-zero (it has to be square
and of full rank).
A matrix that is not invertible is said to be singular.
Reciprocally, a matrix that is invertible is said to be non-singular.
Linear Algebra & Matrices, MfD 2010
Pseudoinverse
In SPM, design matrices are not square (more
lines than columns, especially for fMRI).
The system is said to be overdetermined – there
is not a unique solution, i.e.
there is more than one solution possible.
SPM will use a mathematical trick called the
pseudoinverse, which is an approximation used in
overdetermined systems, where the solution is
constrained to be the one where the  values that
are minimum.
Linear Algebra & Matrices, MfD 2010
Part III
How are matrices relevant
to fMRI data?
Linear Algebra & Matrices, MfD 2010
Image time-series
Realignment
Spatial filter
Design matrix
Smoothing
General Linear Model
Statistical Parametric Map
Statistical
Inference
Normalisation
Anatomical
reference
Parameter estimates
Linear Algebra & Matrices, MfD 2010
RFT
p <0.05
Voxel-wise time series analysis
Model
specification
Time
Parameter
estimation
Hypothesis
Statistic
BOLD signal
single voxel
time series
Linear Algebra & Matrices, MfD 2010
SPM
How are matrices relevant to fMRI data?
GLM equation

m
N of scans
3
4
=
5
+
6
7
8
9
Y
=
X
Linear Algebra & Matrices, MfD 2010
´

+
e
How are matrices relevant to fMRI data?
Response variable
A single voxel sampled at successive
time points.
Each voxel is considered as
independent observation.
Y
Linear Algebra & Matrices, MfD 2010
Y
Ti
Time
me
e.g BOLD signal at a particular
voxel
Preprocessing
...
Intens
ity
Y=
X.β +ε
How are matrices relevant to fMRI data?
Explanatory variables

m
3
4
5
6
– These are assumed to be
measured without error.
– May be continuous;
– May be dummy,
indicating levels of an
experimental factor.
7
8
9
X
´

Linear Algebra & Matrices, MfD 2010
Solve equation for β – tells us
how much of the BOLD signal is
explained by X
Y=
X.β +ε
In Practice
• Estimate MAGNITUDE of signal changes
• MR INTENSITY levels for each voxel at
various time points
• Relationship between experiment and
voxel changes are established
• Calculation and notation require linear
algebra and matrices manipulations
Linear Algebra & Matrices, MfD 2010
Summary
• SPM builds up data as a matrix.
• Manipulation of matrices enables unknown
values to be calculated.
Y
=
X
. β
+
ε
Observed = Predictors * Parameters + Error
BOLD
= Design Matrix * Betas + Error
Linear Algebra & Matrices, MfD 2010
References
• SPM course http://www.fil.ion.ucl.ac.uk/spm/course/
• Web Guides
http://mathworld.wolfram.com/LinearAlgebra.html
http://www.maths.surrey.ac.uk/explore/emmaspages/option1.h
tml
http://www.inf.ed.ac.uk/teaching/courses/fmcs1/
(Formal Modelling in Cognitive Science course)
• http://www.wikipedia.org
• Previous MfD slides
Linear Algebra & Matrices, MfD 2010
ANY QUESTIONS ?