Download Lecture 3 Nearest Neighbor Algorithms

Lecture 6
Matrix Operations
and Gaussian Elimination for
Solving Linear Systems
Shang-Hua Teng
Matrix
(Uniform Representation for Any Dimension)
• An m by n matrix is a rectangular table of
mn numbers
 a1,1
a
2 ,1

A
 

am ,1
a1, 2
a2 , 2

am , 2
a1,n 
... a2,n 
...  

... am ,n 
...
Sometime we write A(i, j )  ai , j
Matrix
(Uniform Representation for Any Dimension)
 a1,1 a1, 2
a
a2 , 2
2 ,1

A
 


am,1 am, 2
... a1,n 

... a2,n 
...  

... am,n 
• Can be viewed as m row vectors in n
dimensions
Matrix
(Uniform Representation for Any Dimension)
 a1,1 a1, 2
a
a2 , 2
2 ,1

A
 


am,1 am, 2
... a1,n 

... a2,n 
...  

... am,n 
• Or can be viewed as n column vectors in m
dimensions
Squared Matrix
• An n by n matrix is a squared table of n2
numbers
 a1,1
a
2 ,1

A
 

an,1
a1, 2 ... a1,n 

a2, 2 ... a2,n 
 ...  

an , 2 ... an ,n 
Some Special Squared Matrices
• All zeros
0
matrix
0
0(n, m)  


0
0
0

0
...
...
...
...
0
0


0
• Identity matrix
1
0
I  I (n, n)  


0
0
1

0
...
...
...
...
0
0


1
Matrix Operations
• Addition
• Scalar multiplication
• Multiplication
1. Matrix Addition:
a11 a12 a1n 
a a  a 
2n 
A   21 22
,






a
a

a
mn 
 m1 m 2
b11b12b1n 
b b b 
B   21 22 2 n ,
 
 


b
b

b
 m1 m 2 mn 
a11  b11 a12  b12  a1n  b1n 
a  b a  b  a  b 
2n
2n 
A  B   21 21 22 22
 





a

b

a

b
mn
mn 
 m1 m1
Matrices have to have the same dimensions
What is the complexity?
2. Scalar Multiplication:
a11a12a1n    a11   a12    a1n 
a a a    a   a    a 
21
22
2n 
  A     21 22 2 n  
 
 





 

am1 am 2amn    am1   am 2    amn 
What is the complexity?
3. Matrix Multiplication
b11b12b1 p 
a11a12a1n 


a a a 
b
b

b
21 22
2p
A   21 22 2 n ,B  
,

 


 





bn1bn 2bnp 
am1 am 2amn 
n a1i  bi1 n a1i  bi 2  n a1i  bip 
i 1
i 1
 i 1

n
n
n


a

b
a

b

a

b



2
i
i
1
2
i
i
2
2
i
ip
i 1
i 1

A  B   i 1
 



 n

n

i =1 ami  bi1
i 1 ami  bip 
Two matrices have to be conformal
What is the complexity?
Matrix Multiplication
b11b12b1 p 
a11a12a1n 


a a a 
b
b

b
21 22
2p
A   21 22 2 n ,B  
,

 


 





bn1bn 2bnp 
am1 am 2amn 



A B  



(row i of A)  (column j of B)
Two matrices have to be conformal







The Laws of Matrix Operations
•
•
•
•
•
•
•
A + B = B + A (commutative)
c(A+B) = cA + c+B (distributive)
A + (B + C) = (A + B) + C (associative)
C(A+B) = CA + CB (distributive from left)
(A+B)C = AC+BC (distributive from right)
A(BC) = (AB)C (associative)
But in general:
AB  BA
Counter Example
0 0  0 1   0 0 
AB  





1 0 0 0 0 1
but
0 1 0 0 1 0
BA  





0 0 1 0 0 0
Special Matrices
• Identity matrix I
– IA = AI = A
• Square Matrix A
Ap  
AAA
A



p
A A   A
A   A
p
p q
q
pq
pq
Elimination: Method for Solving
Linear Systems
• Linear Systems == System of Linear Equations
• Elimination:
– Multiply the LHS and RHS of an equation by a
nonzero constant results the same equations
x : f ( x)  g( x)  x : f ( x)  g( x),   0
– Adding the LHSs and RHSs of two equations does
not change the solution
x : f1 ( x)  g1 ( x); f2 ( x)  g2 ( x)  x : f1 ( x)  g1 ( x); f1 ( x)  f2 ( x)  g1 ( x)  g2 ( x)
Elimination in 2D
x  2y  1
3x  2 y  11
• Multiply the first equation by 3 and subtracts from
the second equation (to eliminate x)
x  2y  1
0
8y  8
• The two systems have the same solution
• The second system is easy to solve
Geometry of Elimination
x  2y  1
3x  2 y  11
3x  2 y  11
(3,1)
8y = 8
x  2y  1
Reduce to a
1-dimensional problem.
x  2y  1
0
8y  8
Upper Triangular Systems and
Back Substitution
x  2y  1
0
8y  8
• Back substitution
– From the second equation y = 1
– Substitute the value of y to the first equation to obtain
x-2=1
– Solve it we have: x = 3
• So the solution is (3,1)
How Much to Multiply before
Subtracting
• Pivot: first nonzero in the row that does the
elimination
• Multiplier: (entry to eliminate) divided by
(pivot)
x  2y  1
3x  2 y  11
Multiply: = 3/1
How Much to Multiply before
Subtracting
• Pivot: first nonzero in the row that does the
elimination
• Multiplier: (entry to eliminate) divided by
(pivot)
2 x  4 y  2 The pivots are on the
of the
3 x  2 y  11 diagonal
triangle after the
Multiply: = 3/2
2x  4 y  2
0
8y  8
elimination
Breakdown of Elimination
• What is the pivot is zero == one can’t divide
by zero!!!!
x  2y  1
3x  6 y  11
Eliminate x:
x  2y  1
0y  8
No Solution!!!!: this system has no second pivot
Geometric Intuition
(Row Pictures)
(3,1)
8y = 8
x  2y  1
3x  6 y  11
• Two parallel lines never intersect
Geometric Intuition
(Column Picture)
1
11
 
x  2y  1
3x  6 y  11
1
3
 
 2
 6
 
Two column vectors are co-linear!!!!
Geometric Intuition
Geometric degeneracy cause failure in
elimination!
Failure in Elimination May
Indicate Infinitely Many Solutions
x  2y  1
3x  6 y  3
x  2y  1
0y  0
• y is free, can be number!
• Geometric Intuition (row picture): The two
line are the same
• Geometric Intuition (column picture): all
three column vectors are co-linear
Failure in Elimination
(Temporary and can be Fixed)
0x  2 y  4
3x  2 y  5
• First pivot position contains zero
• Exchange with the second equation
2x  2 y  5
2y  4
Can be solved by backward substitution!
Singular Systems versus
Non-Singular Systems
• A singular system has no solution or
infinitely many solution
– Row Picture: two line are parallel or the same
– Column Picture: Two column vectors are colinear
• A non-singular system has a unique solution
– Row Picture: two non-parallel lines
– Column Picture: two non-colinear column
vectors
Gaussian Elimination in 3D
2x
 4 y  2z  2
4 x  9 y  3z  8
 2 x  3 y  7 z  10
• Using the first pivot to eliminate x from the
next two equations
Gaussian Elimination in 3D
2x  4 y  2z  2
y  z  4
y  5 z  12
• Using the second pivot to eliminate y from
the third equation
Gaussian Elimination in 3D
2x  4 y  2z  2
y 
z  4
4z  8
• Using the second pivot to eliminate y from
the third equation
Now We Have a Triangular System
2x  4 y  2z  2
y 
z  4
4z  8
• From the last equation, we have
Backward Substitution
2x  4 y  2z  2
y 
z  4
z  2
• And substitute z to the first two equations
Backward Substitution
2x  4 y  4  2
y  2  4
z  2
• We can solve y
Backward Substitution
2x  4 y  4  2
y
 2
z  2
• Substitute to the first equation
Backward Substitution
2x  8  4  2
y
 2
z  2
• We can solve the first equation
Backward Substitution
 1
x
y
 2
z  2
• We can solve the first equation
Generalization
• How to generalize to higher dimensions?
• What is the complexity of the algorithm?
• Answer:
• Express Elimination with Matrices
Step 1
Build Augmented Matrix
2x
 4 y  2z  2
4 x  9 y  3z  8
 2 x  3 y  7 z  10
Ax = b
[A b]
 2 4 2 2
A b   4 9  3 8 
 2  3 7 10
Pivot 1: The elimination of column 1
 2 4 2 2
 4 9 3 8 


 2  3 7 10
 2 4 2 2
 0 1 1 4


 2  3 7 10
2 4  2 2 
0 1 1 4 


0 1 5 12
 
 2
 
 1 
Pivot 2: The elimination of column 2
2 4  2 2 
0 1 1 4 


0 1 5 12
 2 4  2 2
0 1 1 4


0 0 4 8
Upper triangular matrix
 
 
 
 1
Backward Substitution 1: from the
last column to the first
Upper triangular matrix
 2 4  2 2
0 1 1 4


0 0 4 8
1 0 0  1
0 1 0 2 


0 0 1 2 
 2 4  2 2
0 1 1 4


0 0 1 2
 2 0 0  2
0 1 0 2 


0 0 1 2 
 2 4  2 2
0 1 0 2


0 0 1 2
2 4 0 6
0 1 0 2


0 0 1 2
Expressing Elimination by
Matrix Multiplication
Elementary or Elimination Matrix
Ei , j
• The elementary or elimination matrix Ei , j
That subtracts a multiple l of row j from row i
can be obtained from the identity entry by
adding (-l) in the i,j position
 1 0 0
E3,1   0 1 0
 l 0 1
Elementary or Elimination Matrix
 a1,1 a1, 2 a1,3   1 0 0  a1,1 a1, 2 a1,3 

 



E3,1 a2,1 a2, 2 a2,3    0 1 0 a2,1 a2, 2 a2,3 
 a3,1 a3, 2 a3,3   l 0 1  a3,1 a3, 2 a3,3 



a1,1
a1, 2
a1,3


a2,1
a2 , 2
a2 , 3


 la1,1  a3,1  la1, 2  a3, 2  la1,3  a3,3 
Pivot 1: The elimination of column 1
 
 2
 
 1 
 2 4 2 2
 4 9 3 8 


 2  3 7 10
2 4  2 2 
0 1 1 4 


0 1 5 12
Elimination matrix
 1 0 0  2 4  2 2 
  2 1 0  4 9  3 8  



 0 0 1  2  3 7 10
 2 4 2 2
 0 1 1 4


 2  3 7 10
1 0 0  2 4  2 2  2 4  2 2 
0 1 0  0 1 1 4   0 1 1 4 


 

1 0 1  2  3 7 10 0 1 5 12
The Product of Elimination Matrices
1 0 0  1 0 0  1 0 0
0 1 0  2 1 0   2 1 0


 

1 0 1  0 0 1  1 0 1
0 0
1 0 0  1 0 0  1
0 1 0  2 1 0   2 1 0


 

0  1 1  1 0 1  1  1 1