Download Document

Round-off Errors
1
Key Concepts
• Round-off / Chopping Errors
• Recognize how floating point arithmetic
operations can introduce and amplify
round-off errors
• What can be done to reduce the effect of
round-off errors
2
There are discrete points
on the number lines that
can be represented by our
computer.
How about the space
between ?
3
Implication of FP representations
• Only limited range of quantities may be
represented.
– Overflow and underflow
• Only a finite number of quantities within
the range may be represented.
– round-off errors or chopping errors
4
Round-off / Chopping Errors
(Error Bounds Analysis)
Let
z be a real number we want to represent in a
computer, and
fl(z) be the representation of z in the computer.
What is the largest possible value of
z  fl ( z )
?
z
i.e., in the worst case, how much data are we losing due to round-off or
chopping errors?
5
Chopping Errors (Error Bounds Analysis)
Suppose the mantissa can only support n digits.
z  0.a1a2 an an 1an 2    e ,
a1  0
fl( z )  0.a1a2 an    e
Thus the absolute and relative chopping errors are
e
en
z  fl( z )  (0.00
...
0
a
a

)



(
0
.
a
a

)


n 1 n  2

 n 1 n 2 
n zeroes
(0.00...0an 1an 2 )    e
z  fl( z )

z
(0.a1a2 an an 1an 2 )    e
Suppose ß = 10 (base 10), what are the values of ai such
that the errors are the largest?
6
Chopping Errors (Error Bounds Analysis)
Because
0.an1an2an3   1
z  fl( z )  0.an 1an  2    en   en
z  fl( z)
  en
z  fl( z )
0.00...0an 1an  2    e

z
0.a1a2  an an 1an  2    e
 en

0.a1a2  an an 1an  2    e
 en

e
0.100000
a
a



 n 1 n  2
n digits
 en
 en
e  n ( e 1)
1 n






0.1   e  1   e
z  fl ( z )
z
  1 n
7
Round-off Errors (Error Bounds Analysis)
z    0.a1a2  an an1    e ,
a1  0
  1 (sign )   base e  exponent

  (0.a1a2  an )    e

fl ( z )  

e


[(
0
.
a
a

a
)

(
0
.
00
...
01
)
]


1 2
n 



 

 n
0  a n 1 

2
Round down

2
 a n 1  
Round up
fl(z) is the rounded value of z
8
Round-off Errors (Error Bounds Analysis)
Absolute error of fl(z)
When rounding down
z  fl( z )
z  fl( z )
an 1
   0.000an 1an  2 an 3    e
   0.an 1an  2 an 3    en
 0.an 1an  2 an 3    en

1
  (.an 1 )  
2
2
z  fl ( z )
1 en
 
2
z  fl ( z )
1 en
 
2
Similarly, when rounding up
i.e., when

2
 an 1  
9
Round-off Errors (Error Bounds Analysis)
Relative error of fl(z)
1 en
z  fl ( z ) 

2
z  fl ( z )
1  n

z
2 z  e
1
 n

because z  (.a1a2 )    e
2 (.a1a2 ) 
1  n

because (.a1 )   (0.1) 
2 (.1) 
1  n

z  fl ( z )
1 1 n
2  1


z
2
10
Summary of Error Bounds Analysis
Chopping Errors
Round-off errors
Absolute
z  fl( z)  
1 en
z  fl( z )  
2
Relative
z  fl( z )
  1n
z
β
n
e n
z  fl( z ) 1 1n
 
z
2
base
# of significant digits or # of digits in the mantissa
Regardless of chopping or round-off is used to round the
numbers, the absolute errors may increase as the
numbers grow in magnitude but the relative errors are
bounded by the same magnitude.
11
Machine Epsilon
Relative chopping error
z  fl( z )
z
Relative round-off error
z  fl( z )
z
eps is known as the
machine epsilon – the
smallest number such that
epsilon = 1;
while (1 + epsilon > 1)
epsilon = epsilon / 2;
epsilon = epsilon * 2;
1 + eps > 1
  1n  epsChopping

1 1n
  epsRound-off
2
Algorithm to compute machine epsilon
12
Propagation of Errors
• Each number or value of a variable is represented
with error
xT  x A  E x
yT  y A  E y
• These errors (Ex and Ey) are carried over to the
result of every arithmetic operation (+, -, x, ⌯)
• How much error is propagated to the result of
each arithmetic operation?
13
Example #1
xT  0.12346  103
yT  0.45623  101
Assume 4 decimal mantissa with rounding are used
x A  0.1235  103
xA  y A
y A  0.4562  101
 0.1235 103  0.4562 101
 0.128062 103
fl ( x A  y A )
 0.128110 (Final value after
3
round-off)
How many types of errors and how much errors
are introduced to the final value?
14
Example #1
Propagated Error: (xT + yT) - (xA + yA) = Ex+Ey
1
3
y

0
.
45623

10
xT  0.12346  10
T
x A  0.1235  10
y A  0.4562  101
3
E x  0.4000  10
1
E y  0.3000  10 3
Propagated Error = -0.4000x10-1 + 0.3000x10-3
15
Example #1
Rounding Error:
xA  yA
 0.1235  103  0.4562  101  0.128062  103
fl( x A  y A )  0.1281  103
( x A  y A )  fl( x A  y A )
 0.1281  103  0.128062  103
 0.3800  10 1
16
Example #1
Finally, the total error is
( xT  yT )  fl( x A  y A )

( 0.12346  103  0.45623  101 )  0.1281  103

0.1280223  103  0.1281  103

 0.7770  101

1
3
1

0
.
4000

10

0
.
3000

10


0
.
3800

10
 

propagated error
rounding error
The total error is the sum of the propagated error and
rounding error
17
Propagation of Errors (In General)
xT  x A  E x
yT  y A  E y
• Let  be the operation between xT and yT
•  can be any of +, -, x, ⌯
• Let * be the corresponding operation
carried out by the computer
– Note: xA  yA ≠ xA * yA
18
Propagation of Errors (In General)
Error between the true result and the computed
result is
(xT  yT ) – (xA * yA) =
(xT  yT – xA  yA) + (xA  yA – xA * yA)
Errors in x and y propagated
by the operation
Rounding error of the result
|xA  yA – xA * yA| = fl(xA  yA) ≤ |xA  yA | x eps
19
Analysis of Propagated Errors
Addition and Subtraction
Addition
( xT  yT )  ( x A  y A )  ( x A  Ex  y A  E y )  ( x A  y A )
 Ex  E y
Subtraction
( xT  yT )  ( x A  y A )  ( x A  Ex  y A  E y )  ( x A  y A )
 Ex  E y
Note : E x  E y or E x  E y  E x  E y
20
Propagated Errors – Multiplication
 x y
( xT  yT )  ( x A  y A )

xT yT
xT yT  ( xT  E x )  ( yT  E y )

xT yT
yT E x  xT E y  E x E y

xT yT
Ex E y Ex E y
Very
small
and
can

 
be neglected
xT yT xT yT
 x y
21
Propagated Errors – Division
 x/ y
xT x A
xA
xA

yT y A
yA
xT

 1
 1
yT  y A
xT
xT
yA
y 
yT
yT
yT
yT
yA
1  x
  1  y
 1
yT
1  y
x y

if εy is small and negligible
1  y
 x y
22
Example #2
Effects of rounding errors in arithmetic manipulations
• Assuming 4-digit decimal mantissa
• Round-off in simple multiplication or division
3333  111  369963
(EXACT)
Results by the computer:
(0.3333  10 4 )  (0.1110  10 3 )
 0.369963  10 6
 0.3700  10 6
(Rounding)
 370000
(Result)
23
Danger of adding/subtracting a small number to/from
a large number
8000  0.3
 0.8000  104  0.00003  104
 0.80003  104
 0.8000  104
 8000
(Rounding)
Possible workarounds:
1) Sort the numbers by magnitude (if they have the same signs) and
add the numbers in increasing order
2) Reformulate the formula algebraically
24
Associativity not necessarily hold for floating point
addition (or multiplication)
a  0.8567  100 ,
b  0.1325  104 ,
c  0.1325  104
a  (b  c )  0
? .8567  100
( a  b)  c  0
? .1000  101
The two answers are NOT the same!
Note: In this example, if we simply sort the numbers by
magnitude and add the number in increasing order, we
actually get worse answer!
Better approach is analyze the problem algebraically.
25
Subtraction of two close numbers
0.3641  10 2

0.2686  10 2
0.0955  10 2
The result will be normalized into 0.9550 x 101
However, note that the zero added to the end of the mantissa
is not significant.
Note: 0.9550 x 101 implies the error is about ± 0.00005 x 101 but
the actual error could be as big as ± 0.00005 x 102
26
Subtractive Cancellation – Subtraction of two very
close numbers
xT
 0.5764 
1
2
 10 4
yT
 0.5763 
1
2
 10 4
xT  y T
 0.0001  0.0001
The error bound is just as large as the estimation of the result!
Subtraction of nearly equal numbers are major cause of errors!
Avoid subtractive cancellation whenever possible.
27
Avoiding Subtractive Cancellations
Example 1: When x is large, compute
f ( x) 
x 1 
x
Is there a way to reduce the errors assuming that we are using
the same number of bits to represent numbers?
Answer: One possible solution is via rationalization
f ( x) 



x 1 
x 
1
x 1 
x
x 1 
x 1 
x
x
28
Subtraction of nearly equal numbers
Example 2: Compute the roots of ax2 + bx + c = 0 using
x
b
b 2  4ac
2a
when
b 2  4ac
Solve x2 – 26x + 1 = 0
(1)
T
x
(2)
T
x


26 
262  4
 13 
2
168
26 
262  4
 13 
2
168
29
Example 2 (continue)
Assume 5 decimal mantissa,
x A(1)  25.961
Since
x A( 2 )  13.000  12.961  0.039
E x (1)  E x ( 2 )  0.0005
xT(1)  25.961,
x
x
(1)
(2)
168  12.961
xT( 2 )  0.0385186
0.0005
0.0005
5

 1.9  10 ,  x (1) 
 1.3  102
25.961
0.0385186
  x (1) implies that one solution is more accurate
than the other one.
30
Example 2 (continue)
Alternatively, a better solution is
( 2)
xA
 13 

168  13 
1
13 
168

168 
13 
13 
168
168
1
 0.038519
25.961
with  x( 2 )   25.1961   x(1)
x
i.e., instead of computing
we use x 
b 
4ac
2a(b 
b  4ac )
2

b 2  4ac
2a
2c
b
b 2  4ac
as the solution for the second root
31
• Note: This formula does NOT give more
accurate result in ALL cases.
• We have to be careful when writing
numerical programs.
• A prior estimation of the answer, and the
corresponding error, is needed first. If the
error is large, we have to use alternative
methods to compute the solution.
32
Assignment 1 (Problem 1)
Given
f ( x) 
1

x
1
x 1
Assume 3 decimal mantissa with rounding
(a) Evaluate f(1000) directly.
(b) Evaluate f(1000) as accurate as possible using
an alternative approach.
(c) Find the relative error of f(1000) in part (a) and
(b).
33
Propagation of Errors in a Series
Let the series be
S 
m
x
i
i
Is there any difference between adding
(((x1 + x2) +x3) +x4) +…+xm and
(((xm + xm-1) +xm-2) +xm-3) +…+x1
34
Example:
#include <stdio.h>
int main() {
float sumx, x;
float sumy, y;
double sumz, z;
int i;
sumx = 0.0;
sumy = 0.0;
sumz = 0.0;
x = 1.0;
y = 0.00001;
z = 0.00001;
for (i
sumx
sumy
sumz
}
=
=
=
=
0; i
sumx
sumy
sumz
<
+
+
+
100000; i++) {
x;
y;
z;
printf("%sumx = %f\n", sumx);
printf("%sumy = %f\n", sumy);
printf("%sumz = %f\n", sumz);
return 0;
}
Output:
sumx = 100000.000000
sumy = 1.000990
sumz = 0.99999999999808375506
35
Exercise
Discuss to what extent
(a + b)c = ac + bc
is violated in machine arithmetic.
36
Example: Evaluate ex as
2
3
n
x
x
x
e x  1  x    ...   ...
2! 3!
n!
#include <stdio.h>
#include <math.h>
int main() {
float x = 10, sum = 1, term = 1, temp = 0;
int i = 0;
while (temp != sum)
i++;
term = term * x /
temp = sum;
sum = sum + term;
printf("%2d %-12f
}
printf("exact value
return 0;
}
{
i;
%-14f\n", i, term, sum);
= %f\n", exp((double)x));
37
Output (when x = 10)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
term
sum
10.000000
50.000000
166.666672
416.666687
833.333374
1388.889038
1984.127197
2480.158936
2755.732178
2755.732178
2505.211182
2087.676025
1605.904541
1147.074585
764.716431
477.947754
11.000000
61.000000
227.666672
644.333374
1477.666748
2866.555664
4850.682617
7330.841797
10086.574219
12842.306641
15347.517578
17435.193359
19041.097656
20188.171875
20952.888672
21430.835938
17 281.145752
21711.982422
18 156.192078
21868.173828
19 82.206360
21950.380859
20 41.103180
21991.484375
21 19.572943
22011.056641
22 8.896792
22019.953125
23 3.868171
22023.822266
24 1.611738
22025.433594
25 0.644695
22026.078125
26 0.247960
22026.326172
27 0.091837
22026.417969
28 0.032799
22026.451172
29 0.011310
22026.462891
30 0.003770
22026.466797
31 0.001216
22026.468750
32 0.000380
22026.468750
exact value = 22026.465795
38
Example: Evaluate ex as
2
3
n
x
x
x
e x  1  x    ...   ...
2! 3!
n!
#include <stdio.h>
#include <math.h>
int main() {
float x = 10, sum = 1, term = 1, temp = 0;
int i = 0;
while (temp != sum)
i++;
term = term * x /
temp = sum;
sum = sum + term;
printf("%2d %-12f
}
printf("exact value
return 0;
}
{
i;
Arithmetic operations
that introduce errors
%-14f\n", i, term, sum);
= %f\n", exp((double)x));
39
Output (when x = -10)
1
2
3
4
5
6
7
8
9
10
11
12
13
…
term
sum
-10.000000
50.000000
-166.666672
416.666687
-833.333374
1388.889038
-1984.127197
2480.158936
-2755.732178
2755.732178
-2505.211182
2087.676025
-1605.904541
-9.000000
41.000000
-125.666672
291.000000
-542.333374
846.555664
-1137.571533
1342.587402
-1413.144775
1342.587402
-1162.623779
925.052246
-680.852295
Not just incorrect answer!
We get negative value!
29 -0.011310
-0.002908
30 0.003770
0.000862
31 -0.001216
-0.000354
32 0.000380
0.000026
33 -0.000115
-0.000089
34 0.000034
-0.000055
35 -0.000010
-0.000065
36 0.000003
-0.000062
37 -0.000001
-0.000063
38 0.000000
-0.000063
39 -0.000000
-0.000063
40 0.000000
-0.000063
41 -0.000000
-0.000063
42 0.000000
-0.000063
43 -0.000000
-0.000063
44 0.000000
-0.000063
45 -0.000000
-0.000063
46 0.000000
-0.000063
exact value = 0.000045
40
Errors vs. Number of Arithmetic
Operations
Assume 3-digit mantissa with rounding
(a) Evaluate y = x3 – 3x2 + 4x + 0.21
for x = 2.73
(b) Evaluate y = [(x – 3)x + 4] x + 0.21 for x = 2.73
Compare and discuss the errors obtained in part (a)
and (b).
41
Summary
• Round-off/chopping errors
– Analysis
• Propagation of errors in arithmetic operations
– Analysis and Calculation
• How to minimize propagation of errors
– Avoid adding huge number to small number
– Avoid subtracting numbers that are close
– Minimize the number of arithmetic operations involved
42