Download Lecture 1

Mathematical preliminaries
1
What is numerical analysis and scientific computing?
Numerical analysis is the study of algorithms that use numerical approximation (as opposed to general
symbolic manipulations) for the problems of mathematical analysis. (from wiki)
The scientific computing/computational mathematics
• developing and analyzing numerical techniques for solving algebraic or differential equations
• use computers to analyze/solve scientific/engineering problems.
• based on mathematical modeling of practical problems
2
Calculus
KEY WORDS. Limit, Convergence, Continuous Function, derivative, Intermedia/Mean Value Theorem,
Taylor Expansion, Big O Notation.
GOAL. Understand the Intermedia/Mean Value Theorem; understand the big O notation; be able to
perform the basic Taylor expansion.
Definition 2.1. (Limit) A function f defined on a set X of real numbers has the limit L at x0 , written
lim f (x) = L,
x→x0
if for any given > 0, there exists a real number δ > 0, such that
|f (x) − L| < ,
whenever
x∈X
and
0 < |x − x0 | < δ
Definition 2.2. (Continuous) Let f be a function defined on a set X of real numbers and x0 ∈. Then f is
continuous at x0 if
lim f (x) = f (x0 ).
x→x0
The function f is continuous on the set X if it is continuous at each number in X.
Definition 2.3. (Convergence of sequence) Let {x0 }∞
n=1 be an infinite sequence of real or complex numbers.
The sequence {x0 }∞
n=1 has the limit x (converges to x), if for any > 0, there exists a positive integer
N () such that |x − xn | < , whenever n > N (). The notation
lim xn = x,
n→∞
or xn → x,
as
n → ∞,
means that the sequence {x0 }∞
n=1 converges to x.
Theorem 2.4. (Continuous) If f is a function defined on a set X of real numbers and x0 ∈ X, then the
following statement is equivalent
• f is continuous at x0
• If {xn }∞
n=1 is any sequence in X converging to x0 , then limn→∞ f (xn ) = f (x0 ).
Definition 2.5. (Differentiable) Let f be a function defined in an open interval containing x0 . The function
f is differentiable at x0 if
f (x) − f (x0 )
f 0 (x0 ) = lim
x→x0
x − x0
exists. The number f 0 (x0 ) is called the derivative of f at x0 . A function that has a derivative at each
number in a set X is differentiable on X.
1
Theorem 2.6. (Intermediate Value Theorem.) Let f (x) be a continuous function on the closed interval
[a, b]. Then for every value α in between f (a) and f (b), there is at least one point ξ ∈ [a, b] for which f (ξ) = α.
Theorem 2.7. (Mean Value Theorem.) If f (x) is a continuous function on the closed interval [a, b] and
differentiable on the open interval (a, b), then there is at least one number ξ ∈ (a, b) such that
f (b) − f (a)
= f 0 (ξ).
b−a
Theorem 2.8. (Rolle’s Theorem.) Let f (x) be a continuous function on the closed interval [a, b] and
differentiable on the open interval (a, b). If f (a) = f (b) = 0, then there is at least one number ξ ∈ (a, b) such
that
f 0 (ξ) = 0.
Theorem 2.9. (Taylor’s Theorem.) If f (x) has n + 1 continuous derivatives on [a, b] and c is some point
in [a, b], then for any c and x ∈ [a, b],
f (x) = f (c) + f 0 (c)(x − c) +
where
En+1 =
f (n) (c)(x − c)n
f 00 (c)(x − c)2
+ ... +
+ En+1
2!
n!
f (n+1) (ξ)(x − c)n+1
= O((x − c)n+1 ),
(n + 1)!
(1)
(2)
and ξ lies between c and x and depends on both. Equation (1) is the Taylor expansion of the function f (x)
around the point c.
Notations (the big O notation O) En+1 = O((x − c)n+1 ) is shorthand for the inequality
|En+1 | ≤ C|x − c|n+1 ,
where C is a constant. For example, in equation (2), C can be
f (n+1) (ξ)
(n+1)! .
Remark 2.10. Taylor expansion is useful in estimating the function value around x = c when
1. much is known about the function and its derivatives at x = c (however, little is know around the
neighborhood of x = c)
2. x is close to c, i.e. En+1 = O((x − c)n+1 ) is very small (within our tolerance for error).
Example 2.11. Taylor expand the following function.
1. f (x) = ex around x = 0
Solution. Since (ex )i = ex , for any i = 1, 2, · · ·
ex
f (n) (0)xn
f (n+1) (ξ)xn+1
f 00 (0)x2
+ ... +
+
2!
n!
(n + 1)!
0 2
0 n
ξ n+1
e x
e x
e x
+ ... +
+
2!
n!
(n + 1)!
xn
xn+1
+ ... +
+ eξ
n!
(n + 1)!
n
x
+ ... +
+ O(xn+1 )
n!
=
f (0) + f 0 (0)x +
=
e0 + e0 x +
=
=
x2
2!
x2
1+x+
2!
1+x+
2. sin(x) around x = 0
2
(3)
Solution. Since sin(x)2i+1 = (−1)i cos(x), for any i = 0, 1, 2, · · · , and sin(x)2i = (−1)i sin(x), for any
i = 1, 2, · · ·
sin(x)
f 00 (0)x2
f (n) (0)xn
f (n+1) (ξ)xn+1
+ ... +
+
2!
n!
(n + 1)!
=
f (0) + f 0 (0)x +
=
sin(0) + cos(0)x +
=
−sin(0)x2
sin(n) (x)xn
sin(n+1) (ξ)xn+1
+ ... +
+
2!
n!
(n + 1)!
3
5
2k+1
x
x
x
x−
+
+ · · · + (−1)k
+ O(x2k+3 )
3!
5!
(2k + 1)!
(4)
Reading Assignments/References:
1. About Intermedia Value Theorem: follow the link http://en.wikipedia.org/wiki/Intermediate_
value_theorem.
2. About Mean Value Theorem: follow the link http://en.wikipedia.org/wiki/Mean_value_theorem.
3. About the Big O Notation: follow the link http://en.wikipedia.org/wiki/Big_O_notation.
4. About Taylor Series: follow the link http://en.wikipedia.org/wiki/Taylor_expansion, or Chapter
1.2 of Cheney & Kincaid
Exercise 2.12. If the following statements are true.
1. If f (x) is a continuous function on the closed interval [a, b]. Let
M = max f (x),
a≤x≤b
m = min f (x).
a≤x≤b
Then for every value α ∈ [m, M ], there is at least one point ξ ∈ [a, b] for which
f (ξ) = α.
2. If f (x) be a continuous function on the closed interval [a, b] with f (a)f (b) < 0, then there is at least
one point ξ ∈ [a, b] for which f (ξ) = 0.
3. If f (x) be a continuous function on the closed interval [a, b] and differentiable on the open interval
(a, b). If f (a) = f (b) = 0, then there is at least one number ξ ∈ (a, b) such that f 0 (ξ) = 0.
Exercise 2.13. Taylor Expand the following function
1. cos(x) around x = 0.
2. log(x) around x = 1.
Exercise 2.14. (Big O Notation) In mathematics, big O notation (so called because it uses the symbol O)
describes the limiting behavior of a function for very small or very large arguments, usually in terms of simpler
functions. Although developed as a part of pure mathematics, it is now frequently also used in computational
complexity theory to describe how the size of the input data affects an algorithm’s usage of computational
resources (usually running time or memory). Please see http://en.wikipedia.org/wiki/Big_O_notation
for the formal definition.
By the definition of Big O notation, if the following statements are true.
1. 99x = O(x), as x → 0.
2. 99x = O(x), as x → ∞.
3. 99x2 = O(x), as x → 0.
4. 99x2 = O(x), as x → ∞.
5. 100N 3 = O(N 4 ), as N → ∞.
6. 100N 5 = O(N 4 ), as N → ∞.
3
3
Linear Algebra
KEY WORDS. Matrix, vector, transpose, inverse, determinant, linear system of equations.
GOAL. Be able to perform the basic matrix operation, such as transpose, inverse, product, determinant
etc; be able to represent a linear system of equations in a matrix vector form.
3.1
Matrix
Definition 3.1. (Matrix) A matrix is a rectangular collection of numbers arranged in rows and columns. A
matrix with n rows and m columns is said to be of dimension n × m. The element at the ith row and j th
column is denoted as aij .
Definition 3.2. (Transpose) The transpose of an n × m matrix A, denoted as AT , is an m × n matrix where
aTij = aji .
Definition 3.3. (Matrix Product) Let A be an n × m matrix, and B be an m × p matrix. The product of
A and B is an n × p matrix C whose elements are defined by
cij =
m
X
aik bkj .
k=1

2
Example 3.4. A =  −4
1


−10 −16
AB =  −4 −4  .
11
16

−6
1

0
is a 3 by 2 matrix, B =
2
5
1
3
is a 2 by 2 matrix. The product
Definition 3.5. (Inverse) If A is an n×n matrix. If there exists an n×n matrix B, such that AB = BA = I,
then B is the inverse of A.
Definition 3.6. (Determinant) The determinant of a square n × n matrix A, det(A), is defined recursively
as follows
1. When n = 1, det(A) = a11 .
2. When n > 1,
det(A) =
n
X
(−1)i+j aij mij ,
j=1
for any choice of row i, or
det(A) =
n
X
(−1)i+j aij mij ,
i=1
for any choice of column j, where mij is the determinant of the (n − 1) × (n − 1) matrix obtained by
deleting the ith row and j th column from A.
a11 a12
Example 3.7. The determinant of an 2 × 2 matrix A =
is a11 a22 − a12 a21 . The determinant
a21 a22
1 2
of A =
is −2.
3 4
a22
−a12
a11 a12
1
−1
Exercise 3.8. Check that the inverse of a 2 × 2 matrix A =
is A = det(A)
−a21 a11
a21 a22
2 3
Exercise 3.9. A 2 × 2 matrix A =
. What are AT , det(A), A−1 ?
4 5
4
4
Computer representation of numbers
Definition 4.1. (Floating point representation in decimal system.) In the decimal system, any real
number x can be represented in a normalized floating point form as
x = ±p × 10n = ±(0.d1 d2 d3 ...)10 × 10n .
where d1 6= 0, hence 0.1 ≤ q < 1. n is an integer. The numbers d1 , d2 , ... are decimal digits 0, 1, · · · 9. For
example, 37.2 = 0.372 × 102 .
Remark 4.2. The requirement of d1 6= 0 is to ensure a unique representation of real number. For example,
37.2 can not be represented as 0.0372 × 103 .
What is the smallest unit of information in computer? A bit. Information/numbers in computer chips
are stored by a series of on and off states (bit), representing 0 and 1. Therefore, numbers wants to be
represented in a binary form.
Definition 4.3. (Floating point representation in binary system.) In the binary system, any real
number x can be represented as
x = ±q × 2m = ±(0.b1 b2 b3 ...)2 × 2m ,
where b1 6= 0, hence b1 = 1 and q ≥ 1/2. For example,
(0.1)2 = 2−1 = 0.5,
(1.1)2 = 20 + 2−1 = 1.5,
(0.1)2 × 20 = 0.5,
(0.1)2 × 2−1 = 0.25
(11)2 = 21 + 20 = 3.
(0.1)2 × 23 = 4.
There are only a finite number of bits in computer to represent a number. A single precision floating
point number is stored in 32 bits; a double precision floating point number is stored in 64 bits. The real
numbers that can be represented in a computer are called its machine numbers. Not all real numbers are
machine numbers.
Definition 4.4. (Standard single-precision floating point representation) A machine number in
standard single-precision floating point representation is,
x = (−1)s × (1.f )2 × 2c−127 .
The number is represented by 32-bit.
• The leftmost bit is assigned to s for the sign. s = 0/1 corresponds to +/− respectively
• The next 8 bits are assigned to c. The value of c in representing a floating point number is restricted
by
0 < c < (11111111)2 = 255.
The values 0 and 255 are reserved for special case of 0 and ±∞. The range of representable machine
numbers is mainly determined by the number of bits assigned to c. Therefore, it can be concluded that
the largest single precision number is
2254−127 × (1.f )2 ≈ 2128 ≈ 3.4 × 1038 ;
The smallest single precision number is
21−127 × (1.0)2 = 2−126 ≈ 1.2 × 10−38 .
• The last 23 bits are assigned to f . The value of (1.f )2 is restricted by
1 ≤ (1.f )2 ≤ (1.1....1)2 = 2 − 2−23 .
The significant decimal digits of accuracy is mainly determined by the number of bits assigned to f .
The number of significant decimal digits of a single precision number is approximately 6, as
2−23 = 0.12 × 10−6 .
5
Definition 4.5. (Standard double-precision floating point representation) A machine number in
standard double-precision floating point representation is,
x = (−1)s × 2c−1023 × (1.f )2 .
The number is represented by 64-bit.
• The leftmost bit is assigned to s for the sign. s = 0/1 corresponds to +/− respectively
• The next 11 bits are assigned to c. The value of c in representing a floating point number is restricted
by
0 < c < (11111111111)2 = 2047.
The values 0 and 2047 are reserved for special case of 0 and ±∞.
• The last 52 bits are assigned to f . The value of (1.f )2 is restricted by
1 ≤ (1.f )2 ≤ (1.1....1)2 = 2 − 2−52 .
The significant decimal digits of accuracy is mainly determined by the number of bits assigned to f .
Exercise 4.6. Give an estimate on the largest and smallest double precision number. Given an estimate on
the number of significant decimal digits of a double precision number.
6