Download PDF/Acrobat version

2
CS 211 Spring 1998
Hal Perkins
Lecture for 3 March 1998
Overview — Algorithm Analysis and Complexity.
Concepts — Space and time complexity, problem size,
big O notation.
Algorithm Analysis
• Goal: Measure resource usage of algorithms and data
structures independent of any particular programming
language or computer.
• Resources: Time and Space.
• Measure resource usage as a function of problem size.
• Problem size: Amount of input to be processed,
length of list to be searched or sorted, etc. May well
depend on what information we are trying to discover
about a particular algorithm or data structure.
3
4
Basic Steps
Example: n!
Basic step: an operation whose execution time is
essentially constant regardless of the value(s) or
variable(s) involved. Examples:
• assignment of a scalar value (int, double, …) or
object reference (but not copying entire objects).
• Input or output of a scalar value.
• Method call (just the call; evaluation of arguments
and execution of the method body uses additional
resources).
Store n! in variable fact.
fact = 1;
for (k = 2; k <= n; k++)
fact = fact * k;
To analyze the running time of this loop, pick a way to
measure the problem size, then analyze the running
time as a function of that size.
Problem size: _______________________ .
• Comparison of scalars (==, !=, <, <=, >, >=).
• arithmetic and logical operations on scalar values
(+ - * / < <= != == >= > && || !).
Time complexity is the number of basic steps measured
as a function of the problem size.
Space complexity is measured similarly: the size of a
data structure, number of objects allocated, etc. as a
function of the problem size.
Running time ≈ _________________ .
5
6
Example: Matrix Multiplication
Example: Print Blocks of *'s
Multiply two n x n matrices A and B and store the
product in matrix C.
for (i = 0; i < n; i++)
for (j = 0; j < n; j++) {
C[i][j] = 0.0;
for (k = 0; k < n; k++)
C[i][j] = C[i][j] + A[i][k]*B[k][j];
}
Problem size: _________________ .
Running time ≈ _________________ .
Print n 3 by 3 blocks of '*'s.
for (i = 0; i < n; i++)
for (j = 0; j < 2; j++) {
for (k = 0; k < 2; k++)
System.out.print('*');
System.out.println( );
}
System.out.println( );
}
Problem size: _________________ .
Running time ≈ _________________ .
7
8
Example: Linear Search
Example: Binary Search
Locate x in ordered (sorted) array b[0..nÐ1] using
linear search.
k = 0;
while (k < n && x < b[k])
k++;
found = (k < n && x == b[k]);
Problem size: _________________ .
Locate x in ordered (sorted) array b[0..nÐ1] using
binary search.
L = -1; R = n;
/* inv: b[0..L] < x && b[R..n-1] >= x */
while (L+1 != R) {
mid = (L + R) / 2;
if (b[mid] < x)
L = mid;
else /* b[mid] >= x */
R = mid;
}
found = (R < n && b[R] == x);
Problem size: _________________ .
Running time ≈ _________________ .
Running time ≈ _________________ .
9
10
Asymptotic Complexity
Perspective
We are usually interested in the asymptotic complexity
of an algorithm. This is the time or space required as
the size of the problem becomes large.
Asymptotic complexity, basically ignores low-order
terms and constant multipliers. For large problems,
only the high-order growth rate usually matters.
Example: Suppose an algorithm requires time
T(n) = 6n3 + 2n2 + 42n + 17
for input of size n. As n gets increasingly large, the 6n3
term will eventually dominate the rest. We say that
T(n) is O(n3) (pronounced "order n cubed") because its
growth rate is proportional to n3.
A caution: Sometimes an algorithm with better
asymptotic complexity is not best for problems of the
size we actually wish to solve because of large constant
factors. Constants do matter in real life, but we will
ignore them for now.
n
n log n
n2
3n2
n3
2n
Complexity
n
n log n
n2
3n2
n3
2n
1 second
1000
140
31
18
10
9
1 minute
60,000
4893
244
144
39
15
12
Another Perspective
Big O Notation
Max problem size
on old machine
a
b
c
d
e
f
Max problem size
on new machine
10a
≈10b for large b
3.16c
3.16d
2.15e
f + 3.3
So, although computers continue to get faster, the extra
work that can be done by the faster machine is nowhere
near what we can accomplish by picking an algorithm
with better asymptotic complexity, at least for large
problems.
1 hour
3,600,000
200,000
1897
1096
153
21
*This and the following table adapted from Aho, Hopcroft & Ullman, The
Design and Analysis of Computer Algorithms, Addison-Wesley, 1974.
11
Suppose we replace our computer with one that is 10
times faster. How much larger are the problems we can
solve on the new machine?
Complexity
Suppose that we have a machine that can execute 1,000
steps of an algorithm per second and we have five
algorithms with different time complexities. Here is the
size of the problem that can be solved in a second, a
minute, and an hour, by each algorithm.*
Big O notation is a precise way to characterize
asymptotic running time (or size).
Def. Let f(n) and g(n) be functions. We say that f(n) is
of order g(n), written O(g(n)), if there is a constant
c > 0 such that, for all positive values of n (except
possibly for a finite number of values),
f(n) ≤ c * g(n) .
In other words, eventually the growth of g(n) dominates
that of f(n) as n gets large, ignoring constants.
Example: f(n) = n+5 and g(n) = n . We show that n+5
is O(g(n)) by taking c = 6. Then
f(n) = n+5 ≤ 6*n = 6*g(n) for n > 0.
13
14
Problem: Let f(n)=17n and g(n)=3n2. Show that f(n) is
O(g(n)).
Terminology and Fine Points
Solution: We must find a constant c such that
f(n) ≤ c * g(n)
• If f(n) is of order g(n) according to the definition, then
these are phrases are equivalent:
— The algorithm runs in time proportional to g(n).
for all positive values of n (except, possibly, for some
finite set of values).
—The running time is order g(n).
— The running time is O(g(n)).
Solution:
— It's an order g(n) algorithm.
• The O( ) notation specifies that g(n) dominates f(n),
but it is not a tight bound. If an algorithm is O(n2)
then it is also O(n3) and O(2n). But it is not O(n).
• We usually want to find a tight bound — a function
g(n) that has the smallest growth rate that still
satisfies the definition. That usually gives us more
useful information about the performance of the
algorithm. (e.g., knowing that an algorithm is O(2n)
is not particularly useful.)
15
16
Comparison of Data Structure Implementations
Evaluation
O( ) notation gives us a quantitative way to compare
different implementations of a data structure.
Question: What is the best representation of a sorted
list: an array or a linked list?
Example: Compare implementations of a sorted list
containing n items as a linked list vs. an array.
Answer: It depends.
Question: On what?
Operation
Create empty list
Delete all list elements
Compute length
Search for item
Access kth element in list
Access next list element
Access previous list element
Insert/delete accessed item
Array Linked List
O(1)
O(1)
O(1)
O(n)
O(1)
O(n)
O(log n)
O(n)
O(1)
O(n)
O(1)
O(1)
O(1)
O(n)
O(n)
O(1)
Answer: On which operations are expected to be
performed frequently in the particular application that
uses the sorted list.