Download Lecture 2. Solving Linear Systems

Lecture 2. Solving Linear Systems
As we discussed before, we can solve any system of linear equations by the
method of elimination, which is equivalent to applying a sequence of elementary
row reductions over its augmented matrix. Some terminologies:
Leading entry –the …rst nonzero entry of a row (or column) is called the
leading entry of the row (or column).
Echelon Form–A matrix is called in Echelon form (upper triangle
form) if
(a) All non-zero rows are above any zero-row (row with all entries zero)
(b) For any two rows, the column containing the leading entry of the upper
row is on the left of the column containing the leading entry of the lower
row.
In short, Echelon form displays upper-triangle pattern. In an Echelon form,
leading entries of rows form a stair-shape. For instance (where * indicates a
leading entry of a row),
2
1
Echelon form: 4 0
0
2
0
0
1
8
0
2
3 2
1
3 0
1 8 5; 4 0
0
0 0
0
non-Echelon form: 4 2
0
2
0
0
2
4
0
1 3
1 1
0 4
3
1 3 0
8 1 8 5:
0 0 0
3
0
8 5
0
Reduced Echelon Form –a matrix is called Reduced Echelon form If,
in addition to (a) and (b) above,
(c) all leading entries are 1 and it is the only non-zero entry in each pivot
column.
For instance, the followings are Reduced
2
3 2
1 0 0 3 0
1
4 0 0 1 1 8 5; 4 0
0 0 0 0 0
0
Echelon forms:
3
0 1 0 0
1 1 0 8 5:
0 0 1 0
Pivot –leading entry 1 in the reduced echelon form is called pivot position (or pivot entry.)
Pivot Column – column containing a pivot position is called pivot column.
1
Theorem 2.1. Any matrix can be reduced by a sequence of elementary row
operations to a unique reduced Echelon form.
Theorem 2.2. Elementary row operations do not a¤ect the solution set of
any linear system. Consequently, the solution set of a system is the same as
that of the system whose augmented matrix is in the reduced Echelon form. The
system can be solved from bottom up once it is reduced to an Echelon form.
Gauss-Jordan Algorithm
Gauss-Jordan Algorithm is a classic algorithm of …nding reduced Echelon
form. It consists of several steps described as follows.
Step 1. From the left, …nd the …rst non-zero column (it is a pivot column).
By interchanging two rows if necessary, make sure that the leading entry of the
column is non-zero. In other words, by performing row operations, we end up
with the following matrix (* represents any number)
3
3
2
2
0 ::: 0 a
:::
a
:::
6 0 ::: 0 b
6 b
::: 7
::: 7
7
7
6
6
4 ::: ::: ::: ::: 5 or 4 ::: ::: ::: ::: ::: ::: 5 ; a 6= 0:
0 ::: 0 c
:::
c
:::
Step 2. Perform elementary row operation #3: R1 =a ! R1 ; we arrive at
2
3
1
=a =a :::
6 b
::: 7
7:
6
4 ::: :::
::: ::: 5
c
:::
Then, perform elementary row operation #1 several times (i.e., …rst, R2 bR1 !
R2 ; ..., …nally Rm cR1 ! Rm ), we obtain a matrix
2
3
1
:::
6 0 A B ::: 7
7
6
4 ::: ::: ::: ::: 5 :
0 C
:::
Step 3. Repeat the above two steps for the submatrix obtained from deleting
the …rst row,
2
3
0 A B :::
4 ::: ::: ::: ::: 5 :
0 C
:::
This sub-matrix is then reduced
2
0
6 0
6
4 :::
0
row-equivalently to
3
1 E
:::
0
::: 7
7;
::: ::: ::: ::: 5
0 G
:::
2
and the original matrix is reduced
2
1
6 0
6
4 :::
0
to
3
:::
E ::: 7
7:
::: ::: 5
G :::
W
1
:::
0
By perform row operation: R1 W R2 ! R1 ;
2
1 0 0
6 0 1
6
4 ::: ::: ::: :::
0 0
we have
3
:::
::: 7
7:
::: 5
:::
Step 4. Repeat till we are forced to stop – this …nal matrix is the reduced
Echelon form.
Example 2.1. Find the reduced Echelon form for
3
2
1
2 1 j 0
4 0
2
8 j 8 5
4 5
9 j
9
and then solve the corresponding linear system
8
x1 2x2 + x3 = 0
<
2x2 8x3 = 8 :
:
4x1 + 5x2 + 9x3 = 9
Solution: We perform row operations on the augmented matrix as follows:
2
1
4 0
4
2
2
5
2
3
2
3
0
1
2 1 j 0
8 5 R3 + 4R1 ! R3 4 0 2
8 j 8 5
!
9
0
3 13 j
9
3
2
2 1 j 0
1 0
7 j
1
4 j 4 5 R1 + 2R2 ! R1 4 0 1
4 j
!
3 13 j
9
0
3 13 j
1 j
8 j
9 j
1
R2 =2 ! R2 4 0
! 0
2
1 0
R3 + 3R2 ! R3 4 0 1
!
0 0
2
1 0
R2 + 4R3 ! R2 4 0 1
!
0 0
3
2
7 j 8
1 0
4 j 4 5 R1 + 7R3 ! R1 4 0 1
!
1 j 3
0 0
3
0 j 29
0 j 16 5 :
1 j 3
We obtain the reduced Echelon form
2
3
1 0 0 j 29
4 0 1 0 j 16 5 :
0 0 1 j 3
3
3
8
4 5
9
3
0 j 29
4 j 4 5
1 j 3
The linear system with the above matrix as its augmented matrix is
8
= 29
< x1
x2
= 16 :
:
x3 = 3
Solution: x1 = 29; x2 = 16; x3 = 3; or (29; 16; 3) :
Solution Sets of Linear Systems
As we discussed above, one can solve any system by reducing its augmented
matrix to the reduced Echelon form. In fact, it is not necessary to get the
reduced echelon form - any Echelon form is enough to solve the system. For
instance, we can solve the same system in Example 2.1 as follows. First, we
perform the same row operations to get an Echelon form
2
3
2
3
1
2 1 j 0
1
2 1 j 0
4 0
2
8 j 8 5 R3 + 4R1 ! R3 4 0 2
8 j 8 5
!
4 5
9 j
9
0
3 13 j
9
2
3
2
3
1
2 1 j 0
1
2 1 j 0
4 j 4 5 R3 + 3R2 ! R3 4 0 1
4 j 4 5:
R2 =2 ! R2 4 0 1
!
! 0
3 13 j
9
0 0
1 j 3
Next, we write down the corresponding system
8
< x1 2x2 + x3 = 0
x2 4x3 = 4 :
:
x3 = 3
Finally, we solve the system bottom up: From the last equation, we see
x3 = 3:
Next, we substitute this into the second equation to get
x2
4 3 = 4 =) x2 = 16:
We then substitute x3 = 3 and x2 = 16 into the …rst equation, we obtain
x1
2 16 + 3 = 0 =) x1 = 29:
In summary, to solve a system of linear equations whose augmented matrix
is M: We follow the following steps.
Step 1: Perform elementary row operations to reduced M to its reduced
Echelon form E.
Step 2: In the matrix E; locate pivot positions and pivot columns. The
variable associated with pivot columns are called basic variables. For instance,
for
2
3
1 0
1 j 1
E = 4 0 1 5 j 6 5 ; x1 and x2 are basic variables:
0 0 0 j 0
4
Step 3: Write down the system of linear equations whose augmented matrix
is E: Solve all pivot variables in terms of non-pivot variables. Those non-basic
variables are referred as to free variables or parameters since they can be
chosen freely to generate solutions.
It is possible to solve a linear system using any Echelon form instead of its
reduced Echelon form. To do so, the very last step needs to be modi…ed to
Step 3 : Write down the system of linear equations whose augmented matrix is E: Solve all pivot variables in terms of non-pivot variables by backward
substitutions.
Example 2.2. Consider a linear system whose augmented matrix is reduced
to an Echelon form
2
3
1 3
1 j 1
4 0 1 2 j 2 5:
0 0 0 j 0
Find the describe the solution set.
Solution: Notice that the matrix has two pivot columns that are associated
with variables x1 and x2 : These two variable are basic variables. x3 is the free
variable. We write down the corresponding system
1
+ 3x2 x3 = 1
x2 + 2x3 = 2;
and solve, from bottom up, basic variables in terms of the free variable. In this
case, from the second equation, we solve x2 in terms of x3 :
x2 = 2
2x3 :
Next, we substitute it into the …rst equation
x1 + 3 (2
2x3 )
x3 = 1
and solve for x1 as
x1 = 13
(2
2x3 ) + x3 = 11 + 3x3 :
Here, x3 can be chosen arbitrarily: for any x3 = t; we have a solution
x1 = 11 + 3t
x2 = 2 2t
x3 = t:
Therefore, there are in…nite many solitons. For instance, take t = 0;
x1 = 11; x2 = 2; x3 = 0;
take t = 1
x1 = 14; x2 = 0; x3 = 1:
5
(1)
We call (1) a parametric representation of the solution set, or general solution.
In general, we choose non-pivot variables as parameters and solve pivot variables
in terms of non-pivot variables.
Example 2.3 Solve
x1
2x2 + x3 + x4 = 0
2x2
+ 6x4 = 4
and describe general solution by a parametric representation.
Solution: Its augmented matrix
1
0
2
2
1 1 j 0
0 6 j 4
is already in Echelon form, and x1 and x2 are pivot variables. So we …rst solve
x2 from the second equation as
x2 = 2
3x4 :
Then substitute it into the …rst equation and solve for x1 as
x1
x1 = 2 (2
2 (2
3x4 )
3x4 ) + x3 + x4 = 0
x3
x4 = 4
x3
7x4 :
Set x3 = s; x4 = t; we …nd a parametric representation
x1
x2
x3
x4
=4
=2
=s
= t:
s 7t
3t
Consistency
Not every system has a solution. For instance, consider a system of three
equation with three unknowns. In 3D space, each linear equation represents a
plane. So the system represents the intersection of three planes, which could be
a single point, or a straight line, or a plane (all three planes are identical), or
empty set (no solution).
Theorem 2.3 (Consistency) A system is called consistent if it admits at
least one solution. A system is consistent if and only if the rightmost column
contains NO pivot.
Proof: If the rightmost column is a pivot column, then in its reduced Echelon
form, there is a pivot in the rightmost column:
2
3
1
:::
j
6 0
7
:::
j
6
7
4 0 0 ::: A j b 5 ; A = 0:
0 0 0 0 j 0
6
In other words, there is a row (the last not-all-zero row) having the form
[0; 0; :::; 0; b], where b 6= 0: Accordingly, the very last non-trivial equation is
0 = b; but b 6= 0;
A contradiction.
Example 2.4. In the second example of
2
3
2
0 1
4 j 8
1
4 2
3 2 j 1 5!4 0
5
8 7 j 1
0
the previous section, we have
3
3=2 1 j 1=2
1
4 j 8 5:
0
0 j 5=2
The last column is a pivot column. Thus, it is inconsistent.
Example 2.5. Determine if the system
3x1
3x1
3x2 6x3 + 6x4 + 4x5 = 5
7x2 + 8x3 5x4 + 8x5 = 9
9x2 + 12x3 9x4 + 6x5 = 15
Solution. The augmented matrix is
2
0 3
6 6 4 j
4 3
7 8
5 8 j
3
9 12
9 6 j
We perform following row operations:
2
0 3
6 6 4 j
4 3
7 8
5 8 j
3
9 12
9 6 j
2
3
7 8
6
R2 ! R1 4 0 3
!
3
9 12
2
3
7
R 3 R1 ! R 3 4 0 3
!
0
2
2
3
7
2
R3 + R2 ! R3 4 0 3
3
! 0 0
3
5
9 5
15
5
6
9
8
6
4
8
6
0
8 j
4 j
6 j
5
6
4
3
5
9 5:
15
3
9
5 5
15
8 j
4 j
2 j
5
8 j
6
4 j
0 2=3 j
3
9
5 5
6
3
9
5 5:
We conclude that the linear system is consistent.
When a linear system is consistence, from the above examples, we see that
the system admits a unique solution if and only if there is no free variable.
Theorem 2.4. Consider a linear system whose augmented matrix is M:
There are three possibilities. (1) If the last column of M is a pivot column, then
the system is inconsistent. (2) If the last column of M is a not pivot column
but the rest columns are all pivot, then the system admits a unique solution.
7
(3) If, additional to the last column that is not pivot, there is at least one more
non-pivot column, then the system has in…nite many solutions.
Note that Gauss-Jordan Algorithm is the most basic algorithm. In Matlab
or Mathematica, there are several build-in functions for solving linear systems.
There are many more e¢ cient algorithm for solving linear systems. Here we
introduce another popular algorithm.
LU Decomposition
Another classical method to solve linear system A~x = ~b is so-called LU
decomposition: Find a lower triangle matrix L and a upper triangle matrix U
such that
2
32
3
1 0
0 a11
6 1
6
7
07
7 6 0 a22
7
6
Am n = Lm m Um n = 6 . . .
7
6
7:
.
.
.
.
..
..
. . .. 5 4 ..
4 .. ..
. .. 5
0
0
1
Now the linear system A~x = ~b can be solved by solving two easier systems
L~y = ~b
U~x = ~y :
The …rst system L~y = ~b can be solved from top to bottom, and the second
system can be solved from bottom up.
LU Decomposition Algorithm
1. Reduce A to an echelon form from U by a sequence of type one row
operations (row replacement row operation)
2. Place entries in L such that the same sequence of row operations reduces
L to the identity matrix.
The rigorous justi…cation of this algorithm will be presented later in Lecture
7.
Example 2. Find LU decomposition and
2
2
4
1 5
6 4
5
3
8
A=6
42
5
4 1
6 0
7
3
then solve for A~x = ~b :
3
2 3
2
1
607
17
7 ; ~b = 6 7 :
425
85
1
14
Solution: Obviously, since A is 4 5; L should be 4 4: The …rst step to reduce
A to an echelon form is to replace the …rst column as follows
2
3
2
3
2
4
1 5
2
2 4
1 5
2
2R1 + R2 ! R2 6
6 4
5 3
8 17
1
2
37
7 R1 + R3 ! R3 60 3
7 = A1 :
A=6
42
40
5
4 1
85
9
3
4 10 5
3R1 + R4 ! R4
6 0
7
3
1
4
12
5
! 0 12
8
Therefore,
2
3
2
3
1 0 0 0
1 0 0 0
2R
+
R
!
R
1
2
2
6 2 1 0 07
60 1 0 07
7
6
7
L=6
4 1 a 1 05 R1 + R3 ! R3 40 a 1 05 = L1 :
3R1 + R4 ! R4
3 b c 1
! 0 b c 1
We repeat
2
2
60
A1 = 6
40
0
the process to …nd a; b :
3
2
4
1 5
2
2 4
6
3
1
2
37
3R
+
R
!
R
0
2
3
3
7
6 3
9
3
4 10 5 4R2 + R4 ! R4 40 0
! 0 0
12
4
12
5
and accordingly a = 3; b = 4 because
3
2
2
1 0 0 0
1
60 1 0 07 3R2 + R3 ! R3 60
7
6
L1 = 6
40
3 1 05 4R2 + R4 ! R4 40
! 0
0 4 c 1
Finally, we …nd
2
2
60
A2 = 6
40
0
1
1
0
0
5
2
2
4
so c = 2 since
2
1
60
6
L2 = 4
0
0
0
1
0
0
0
0
1
2
We conclude
2
1
6 2
L=6
41
3
0
1
3
4
Now we solve
2
2
3
2
2
60
37
7 2R3 + R4 ! R4 6
15
! 40
0
7
4
3
0
0
2
3
1
0
6
07
7 2R3 + R4 ! R4 60
05
! 40
0
1
0
1
0
0
3
2
0 0
2
60
0 07
7; U = 6
40
1 05
2 1
0
1
6 2
L~y = 6
41
3
4
3
0
0
1
1
0
0
5
2
2
0
32 3 2 3
0 0 0
y1
1
6y2 7 6 0 7
1 0 07
76 7 = 6 7
3 1 05 4y3 5 4 2 5
4 2 1 y4
14
9
5
2
2
4
3
2
37
7 = A2
15
7
3
0 0
0 07
7 = L2 :
1 05
c 1
0
1
0
0
c:
4
3
0
0
1
1
0
0
1
1
0
0
0
0
1
0
5
2
2
0
3
0
07
7:
05
1
3
2
37
7:
15
5
3
2
37
7
15
5
from top down as
y1
2y1 + y2
y1 3y2 + y3
3y1 + 4y2 + 2y3 + y4
=1
= 0 =) y2 = 2y1 = 2
= 2 =) y3 = 2 y1 + 3y2 = 2 1 + 6 = 7
= 14 =) y4 = 14 + 3y1 4y2 2y3 = 14 + 3
8
14 =
Finally, we solve
2
2
60
6
40
0
4
3
0
0
1
1
0
0
5
2
2
0
32 3 2 3 2
y1
2 x1
6x2 7 6y2 7 6
37
76 7 = 6 7 = 6
1 5 4x3 5 4y3 5 4
y4
x4
5
from bottom up ( x3 is free variable):
x5 =
1
2x4 + x5 = 7 =) x4 =
3x2 + x3 + 2x4
2x1 + 4x2
x3 + 5x4
3
1
27
7
75
5
7
x5
2
2x4 + 3x5
3
2 x3 8 3
x3
=
= 3
3
3
1 4x2 + x3 5x4 + 2x5
2x5 = 1 =) x1 =
2
x3
1 4
3
+ x3 20 2
7
3
=
= x3
2
6
3x5 = 2 =) x2 =
2
=4
x3
9
:
2
Remark: When there is a need to exchange two rows, the LU algorithm
need to modi…ed accordingly.
Homework #2
1. Determine whether each of the following matrices is in reduced Echelon
form or only in Echelon form, or neither. If in Echelon form, indicate all
pivot positions and pivot columns.
2
3
0 1 0 6 0
5
5 0 9 5
(a) 4 0 0 1
0 0 0 0 1 15
2
3
1 1 2 6 0
5
5 1 9 5
(b) 4 0 0 1
0 0 0 0 0 1
3
2
1 1 2 6 0
5 1 5
(c) 4 0 0 1
0 1 0 0 0
10
5:
2. Determine the value(s) of h such that the matrix is the augmented matrix
of a consistent linear system. Then …nd the general solution.
2
3
1 1 2 6 0
4 1 0 1
5 h 5:
3 1 0 16 2
3. Find general solutions for linear systems whose augmented matrices are
given:
(a)
(b)
2
1
4 0
1
7
0
7
2
1
4 0
0
2
0
0
0
1
4
3
1
0
3
A=4 3
9
3
3
2 5:
3
1
0
1
1
2
2
1
2
6
3
2
105
6
4. Find LU Decomposition for
2
3
5
3 5:
7
6
2
2
make sure verify your answer by direct multiplication of LU:
5. For each of the following statements, determine whether it is true or false.
If your answer is true, state your rationale. If false, provide an counterexample (the example contradicting the statement).
(a) A matrix may be row reduced to more than one matrix in reduced
echelon form, using di¤erent sequences of row operations.
(b) A basic variable is a variable corresponding to a pivot columns
(c) If one row in an echelon form of an augmented matrix is [0 0 0 0 0 3 0] ;
then the associated system is inconsistent.
(d) The pivot positions depend on how you do row reduction.
(e) Whenever a system have more than one solution, it must has a free
variable.
11