Download General Linear Systems

Matrix Multiplication
Chapter III – General Linear Systems
By Gokturk Poyrazoglu
The State University of New York at Buffalo – BEST Group – Winter Lecture Series
General Outline
1.
Triangular Systems
2.
The LU Factorization
3.
Pivoting
Triangular Systems Outline
1.
Row oriented Forward Substitution
2.
Row oriented Back Substitution
3.
Column oriented Forward Substitution
4.
Column oriented Back Substitution
5.
Multiple Right-Hand Sides
6.
Nonsquare Triangular Systems Solving
7.
Algebra of Triangular Matrices
Forward Substitution

Consider a lower triangular system

The unknowns are:

General Procedure:
Row Oriented Forward Substitution

L is a square lower triangular matrix

b is a vector

Overwrite b with the solution of Lx=b
Row Oriented Back Substitution

U is a square upper triangular matrix

b is a vector

Solution x:

Overwrite b with the solution of Ux=b
Column Oriented Versions

Example:

Solve for x1 (x1=3); remove from the equations by
Column Oriented Forward Substitution

L is a square lower triangular matrix

b is a vector

Overwrite b with the solution of Lx=b
Column Oriented Back Substitution

U is a square upper triangular matrix

b is a vector

Overwrite b with the solution of Ux=b
Multiple Right-Hand Sides

Consider block matrices L, X, and B

Solve L11X1=B1 for X1.
Remove X1 from block equations as follows :

Nonsquare Triangular System Solving

Consider a block matrix L where m > n

Assume L11 is nonsingular and lower triangular.

Solve L11x=b1 for x

Then x should solve the system 𝐿21 𝐿−111 𝑏1 = 𝑏2

Otherwise, there is no solution to the overall system.
Triangular Matrix Properties

Unit Triangular Matrix:


Triangular matrix with 1’s on the diagonal.
Other Properties

Inverse of an upper triangular is an upper triangular matrix.

The product of two upper triangular is an upper triangular.

Inverse of a unit upper triangular is a unit upper triangular.

The product of two unit upper triangular is an upper triangular.
The LU Factorization

Outline

Background

Gauss transformations

Application

Upper Triangularizing

Existence of LU

Other versions of LU

Rectangular Matrix

Block LU
Background

Example :
Multiply the 1st equation by 2,
Subtract it from the 2nd equation.

Matrix Notation:
Gauss Transformations

Consider a vector V (a stack of 2 block vectors v1 and v2)

Suppose

Define Gauss Transformation matrix
, and define Gauss Vector
Application

Consider a matrix C, and apply an outer product update

Repeat the process;

Algorithm
Upper Triangularizing

Consider a square matrix-A, then

Example:

Note :Diagonal components of A (pivots) should be zero
Existence of LU

If no zero pivots are encountered; then
and
so that

LU factorization does NOT exist if

That means kth pivot is zero.
is singular.
Construction of matrix-L

Consider the example

kth column of L is defined by the multipliers from kth step
Outer Product Point of View

Consider a matrix A as;

Gauss Elimination results:
where
 Hence;
LU Factorization of a Rectangular Matrix

Such matrices L and U exist if
Examples:

Algorithm for the 1st example:

Operation: nr2-r3/3 flops

is nonsingular.
Roundoff Error in Gaussian Elimination
Triangular Solving with Inexact Triangles

If a small pivot is encountered, then we can expect large numbers to be
present in L and U.
Example :

Solution is

in contrast to exact solution
Pivoting

Outline

Interchange Permutations

Partial Pivoting

Complete pivoting

Rook Pivoting
Interchange Permutations

Consider a permutation matrix:

If we multiply matrix A from the left,
rows 1 and 4 interchanged

If we multiply matrix A from the right;
columns 1 and 4 interchanged
Partial Pivoting


Motivation: To guarantee that no multiplier is greater
than 1 in absolute value.
Example : Consider matrix-A, get the largest entry in
the first column to a11
Partial Pivoting


Particular row interchange strategy is called partial
pivoting.
In general :
where U is an upper triangular, and no multiplier is greater
than 1 in absolute value as a consequence of partial
pivoting.
Complete Pivoting



Partial Pivoting was scanning the current subcolumn for
maximal element;
Complete Pivoting scans the current submatrix to find the
largest entry to pivot.
Hence;
Complete Pivoting Properties


Gaussian Elimination with Complete Pivoting is STABLE.
No significant reason to choose Complete pivoting over
Partial pivoting.


Only if matrix A is rank deficient.
In principal, when the pivot of current submatrix is ZERO
at the beginning of step r+1; that indicates that the
rank(A) =r;

In practice, encountering an exactly zero pivot is lesslikely.
Rook Pivoting

Computes the factorization :

Choosing Pivot: Search for an element of current submatrix
that is maximal in BOTH its ROW and COLUMN.
Complete Pivoting
Rook Pivoting Candidate
Comparison

Flops:






Partial pivoting : O(n2)
Complete Pivoting O(n3)
Rook Pivoting O(n2)
Rook has same level of reliability as complete pivoting
and represents same O(n2) overhead as partial pivoting.
Complete Pivoting may be used for rank identification in
principal.
All pivoting methods are stable.
Extra Proof Slides
Proof of LU Factorization Theorem (slide 18)
Proof of Slide 23