Download Chapter 15 Multithreading - Cleveland State University

Developing Efficient Algorithms
IST311 - CIS265/506 Cleveland State University – Prof. Victor Matos
Adapted from: Introduction to Java Programming: Comprehensive Version, Eighth Edition by Y. Daniel Liang
1
Executing Time
Suppose two algorithms perform the same task such as
1. search (linear search vs. binary search) and
2. sorting (selection sort vs. insertion sort).
Which one is better?
One possible approach to answer this question is to implement these
algorithms in Java and run the programs to get execution time. But there are
two problems for this benchmarking approach:
1.
2.
2
2
First, there are many tasks running concurrently on a computer. The execution
time of a particular program is dependent on the system load.
Second, the execution time is dependent on specific input. Consider linear
search and binary search for example. If an element to be searched happens to
be the first in the list, linear search will find the element quicker than binary
search.
Growth Rate
1.
It is very difficult / inaccurate to compare algorithms by measuring
their execution time.
2.
To overcome these problems, a theoretical approach was developed to
analyze algorithms independent of computers and specific input.
This approach provides an approximated description of how a change on
the size of the input affects the number of statements to be executed.
3.
3
3
In this way, you can see how fast an algorithm’s execution time
increases as the input size increases, so you can compare two algorithms
by examining their growth rates.
Analysis of
Algorithms
Google search on ‘Donald Knut’ processed on April 2, 2013
4
4
Growth Rate
Goal :
Estimate the execution ‘time’ of an algorithm in relation to the
input size.
F (N : input size) ⟶ ( N: number of executed statements )
Algorithm_Performance = O( f(n) )
Note
Computer scientists use the Big O notation to abbreviate for “order of magnitude.”
5
5
Big O Notation
Consider linear search. The linear search algorithm compares the key with the
elements in the array sequentially until the key is found or the array is exhausted.
If the key is not in the array, it requires n comparisons for an array of size n.
If the key is in the array, it requires n/2 comparisons on average.
•
•
The algorithm’s execution time is proportional to the size of the array.
If you double the size of the array, you will expect the number of comparisons to double.
The algorithm grows at a linear rate.
6
6
Big O Notation
Observations

The growth rate of the Linear Search algorithm has an order of
magnitude of n.

Using this notation, the complexity of the linear search algorithm is
O(n), pronounced as “order of n” or “linear on n”
7
7
Best, Worst, and Average Cases
For the same input size, an algorithm’s execution time may vary,
depending on the input.
An input that results in the shortest execution time is called the
best-case input and
2. An input that results in the longest execution time is called the
worst-case input.
1.
Worst-case analysis is very useful.You can show that the algorithm will
never be slower than the worst-case.
The analysis is generally conducted for the worst-case.
8
8
Ignoring Multiplicative Constants
The linear search algorithm requires n comparisons in the worst-case and
n/2 comparisons in the average-case. Both cases require O(n) time.
The multiplicative constants have no impact on growth rates.
The growth rate for n/2 or 100n is the same as n, i.e.,
O(n) = O(n/2) = O(100n).
f(n)
n
n/2
100 n
100
100
50
10000
200
200
100
20000
2
2
2
n
9
f( xi+1 ) / f( xi ) ⟶
f(200) / f(100)
f(100) / f(50)
f(20000) / f(10000)
Ignoring Non-Dominating Terms
Consider the algorithm for finding the maximum
number in an array of n elements.
The Big O notation allows you to ignore the non-dominating parts of
the function and highlight its most important component.
For example consider the case: f(n) = 4n + 2;
You should discard the additive and multiplicative factors so f(n)≈n
Therefore, the complexity of this algorithm is O(n).
10
10
Ignoring Non-Dominating Terms
Consider the algorithm for finding the maximum number in an
array of n elements.
int largest = a[0];
int i = 1;
while ( i < a.length ) {
if ( a[i] > largest ) {
largest = a[i];
}
i++;
}
Max number of statements executed in this fragment is:
1 + 1 + 4* N ⟶ f(n) = 4n + 2 ⟶ O(n)
Therefore if the array’s length is n
n =
n =
n =
n =
...
11
11
1000
2000
3000
4000
f(n)
f(n)
f(n)
f(n)
=
=
=
=
4002
8002
12002
16002
Complexity of this
algorithm is O(n).
Useful Mathematical Summations
n(n  1)
1  2  3  ....  (n  1)  n 
2
n 1
a
1
0
1
2
3
( n 1)
n
a  a  a  a  ....  a
a 
a 1
20  21  2 2  23  ....  2( n 1)
12
12
n 1
2
1
n
2 
 2 n 1  1
2 1
Examples: Determining Big-O
Patterns to look for
 Repetition
 Sequence
 Selection
 Logarithm
13
13
Repetition: Simple Loops
executed
n times
for (i = 1; i <= n; i++) {
k = k + 5;
}
constant time
Time Complexity
T(n) = (a constant c) * n = cn = O(n)
Ignore multiplicative constants (e.g., “c”).
14
14
Repetition: Nested Loops
executed
n times
for (i = 1; i <= n; i++) {
for (j = 1; j <= n; j++) {
k = k + i + j;
}
}
constant time
inner loop
executed
n times
Time Complexity
T(n) = (a constant c) * n * n = cn2 = O(n2)
Ignore multiplicative constants (e.g., “c”).
15
15
Repetition: Nested Loops
for (i = 1; i <= n; i++) {
executed
for (j = 1; j <= i; j++) {
inner loop
n times
k = k + i + j;
executed
}
i times
}
constant time
Time Complexity
T(n) = c + 2c + 3c + 4c + … + nc = c n(n+1)/2 =
(c/2)n2 + (c/2)n = O(n2)
quadratic algorithm
Ignore non-dominating terms
16
16
Ignore multiplicative constants
Repetition: Nested Loops
executed
n times
for (i = 1; i <= n; i++) {
for (j = 1; j <= 20; j++) {
k = k + i + j;
}
}
constant time
inner loop
executed
20 times
Time Complexity
T(n) = 20 * c * n = O(n)
Ignore multiplicative constants (e.g., 20*c)
17
17
Sequence
executed
10 times
executed
n times
for (j = 1; j <= 10; j++) {
k = k + 4;
}
for (i = 1; i <= n; i++) {
for (j = 1; j <= 20; j++) {
k = k + i + j;
}
}
Time Complexity
T(n) = c *10 + 20 * c * n = O(n)
18
18
inner loop
executed
20 times
Selection
if (list.contains(e)) {
System.out.println(e);
}
else
for (Object t: list) {
System.out.println(t);
}
O(n)
Constant
Let n be
list.size().
Loop : O(n)
Time Complexity
T(n) = test time (condition) + worst-case (then/ else)
= O(n) + O(n)
= O(n)
19
19
c
Constant Time

The Big O notation estimates the execution time of an algorithm in
relation to the input size.

If the time is not related to the input size, the algorithm is said to take
constant time with the notation O(1).
Example:
A method that retrieves an element at a given index in an array takes
constant time, because it does not grow as the size of the array
increases.
0
1
…
i
…
n-1
Data [i] ⟵ Extract From ( Array_Memory_Loc + (i-1)*cellSize )
20
20
Logarithm: Analyzing Binary Search
Each iteration in the Binary Search algorithm contains a fixed number of
operations, denoted by c (see next listing)
Continue on Lower Half
1
low
21
21
2
3
…
or
n/2
middle
Continue on Upper Half
…
n-2
n-1
n
high
Logarithm: Analyzing Binary Search

Let T(n) denote the time complexity
for a binary search on a list of n
elements.

Without loss of generality, assume n is
a power of 2 ( n = 2k ) and k=log(n).

Since binary search eliminates half
of the input after two comparisons we
have…
n
T ( n)  T ( )  c
2
n
 ( T ( 2 )  c)  c
2
n
 T ( 2 )  2c
2
 ...
2k
 T ( k )  ck
2
 T (1)  c log n
 1  c log n
 O (log n )
22
22
Logarithm: Analyzing Binary Search
public class BinarySearch {
/** Use binary search to find a key in the list */
public static int binarySearch(int[] list, int key) {
int low = 0;
int high = list.length - 1;
while (high >= low) {
int mid = (low + high) / 2;
if (key < list[mid])
high = mid - 1;
else if (key == list[mid])
return mid;
else
low = mid + 1;
}
return -low - 1; // Now high < low
}
23
23
}
Logarithmic Time
Ignoring constants and smaller terms, the complexity of the binary
search algorithm is O( log(n) ).
It is called a logarithmic algorithm.
Note
 The base of the log is 2, but the base does not affect a logarithmic growth
rate, so it can be omitted.
 The logarithmic algorithm grows slowly as the problem size increases. If you
24
24
square the input size, you only double the time for the algorithm.
President Obama & the Bubble Sort
http://www.youtube.com/watch?v=k4RRi_ntQc8
25
Analyzing Selection Sort
SelectionSort (see next listing), finds the largest number in the list and places it
last. It then finds the largest number remaining and places it next to last, and so
on until the list contains only a single number.
4. Exclude last, continue exploration on remaining elements
1
2
3
…
…
n-2
1. Find Largest element
n-1
2. Place largest
element here
temp
3. Exchange if necessary
26
26
n
Analyzing Selection Sort

The number of comparisons is n-1 for the first iteration, n-2 for the
second iteration, and so on.

Let T(n) denote the complexity for selection sort and c denote the total
number of other operations such as assignments and additional
comparisons in each iteration.
T ( n)  [( n  1)  c ]  [( n  2)  c ]  ...  [ 2  c ]  [1  c ]
 [( n  1)  ( n  2)  ...  2  1 ]  nc
 ( n  1) n / 2  nc
27
27
n2
n

  nc
2
2
Ignoring constants and smaller terms, the complexity of the selection
sort algorithm is O(n2).
Analyzing Selection Sort
public class SelectionSort {
public static void selectionSort(double[] list) {
for (int i = 0; i < list.length - 1; i++) {
// Find the minimum in the list[i..list.length-1]
double currentMin = list[i];
int currentMinIndex = i;
for (int j = i + 1; j < list.length; j++) {
if (currentMin > list[j]) {
currentMin = list[j];
currentMinIndex = j;
}
}
// Swap list[i] with list[currentMinIndex] if necessary;
if (currentMinIndex != i) {
list[currentMinIndex] = list[i];
list[i] = currentMin;
}
}
}
28
}
Quadratic Time

An algorithm with the O(n2) time complexity is called a quadratic
algorithm.

A quadratic algorithm grows quickly as the problem size increases. If
you double the input size, the time for the algorithm is quadrupled.

Algorithms with a double-nested loop are often quadratic.
120
100
100
n2
Quadratic
10
n
Linear
81
80
64
60
49
40
36
25
20
29
29
0
16
1
42
9
3
4
5
6
7
8
9
Analyzing Insertion Sort
The insertion sort algorithm (see next listing) sorts a list of values by
repeatedly inserting a new element into a sorted partial array until the whole
array is sorted.
Add 10
10
Add 30
10
30
Add 25
10
30
25
10
25
30
10
25
30
7
10
25
7
30
10
7
25
30
7
10
25
30
Add 7
30
30
Analyzing Insertion Sort

At the kth iteration, to insert an element into an array of size k, it may
take k comparisons to find the insertion position, and k moves to insert
the element.

Let T(n) denote the complexity for insertion sort and c denote the
total number of other operations such as assignments and additional
comparisons in each iteration. So,
T (n)  (2 1  c)  (2  2  c)  ...  (2  (n  1)  c)
 2(1  2  ...  (n  1))  c(n  1)
 n 2  n  cn
 O(n 2 )
31
31
Analyzing Insertion Sort
public static void insertionSort(double[] list) {
for (int i = 1; i < list.length; i++) {
// insert list[i] into a sorted sublist list[0..i-1]
// so that list[0..i] is sorted.
double currentElement = list[i];
int k;
for (k = i - 1; k >= 0 && list[k] > currentElement; k--) {
list[k + 1] = list[k];
}
// Insert the current element into list[k+1]
list[k + 1] = currentElement;
}
}
32
32
Comparing Common Growth Functions
O(1)  O(log n)  O(n)  O(n log n)  O(n 2 )  O(n3 )  O(2n )
33
33
O (1)
Constant time
O (log n )
Logarithmic time
O (n )
Linear time
O (n log n )
Log-linear time
O(n 2 )
Quadratic time
O(n 3 )
Cubic time
O (2n )
Exponential time
Case Study: Fibonacci Numbers
public static long fib(long index) {
if (index == 0) // Base case
return 0;
else if (index == 1) // Base case
return 1;
else // Reduction and recursive calls
return fib(index - 1) + fib(index - 2);
}
Finonacci series: 0 1 1 2 3 5 8 13 21 34 55 89…
indices: 0 1 2 3 4 5 6 7
8
9
fib(0) = 0;
fib(n)
34
34
fib(1) = 1;
fib(n) = fib(n -1) + fib(n -2); n >=2
10 11
Complexity for Recursive Fibonacci
Numbers
Since
T ( n )  T ( n  1)  T ( n  2)  c
 2T ( n  1)  c
 2( 2T ( n  2)  c )  c
 2 T ( n  2)  2c  c
...
2
n 1
 2 T (1)  2
n 2
c  ...  2c  c
n 1
n 2
n 1
n 1
 2 T (1)  ( 2
 2 T (1)  ( 2
 ...  2  1)c
 1)c
and T (n )  T (n  1)  T (n  2)  c
 T ( n  2)  T ( n  3)  c  T ( n  2)  c
 2T ( n  2)  2c
 2( 2T ( n  4)  2c )  2c
 2 2 T ( n  2  2)  2 2 c  2c
 2 3 T ( n  2  2  2)  2 3 c  2 2 c  2c
 2n / 2 T (1)  2n / 2 c  ...  23 c  22 c  2c
 2 n 1 c  ( 2 n 2  ...  2  1)c
 2n / 2 c  2n / 2 c  ...  23 c  22 c  2c
 O (2n )
 O(2n )
Therefore, the recursive Fibonacci method takes
. n
O(2 )
35
35
Case Study: Non-recursive version of Fibonacci Numbers
public
long
long
long
static long fib(long n) {
f0 = 0; // For fib(0)
f1 = 1; // For fib(1)
f2 = 1; // For fib(2)
if (n == 0)
return f0;
else if (n == 1)
return f1;
else if (n == 2)
return f2;
for (int i = 3; i <= n; i++) {
f0 = f1;
f1 = f2;
f2 = f0 + f1;
}
return f2;
}
36
36
Obviously, the complexity
of this new algorithm is
O(n)
This is a tremendous
improvement over the
recursive algorithm.
Practical Considerations
The big O notation provides a good theoretical estimate of algorithm
efficiency.
However, two algorithms of the same time complexity are not necessarily
equally efficient.

As shown in the preceding example, both gcd algorithmsVersion_1 and
Version_2B have the same complexity,

37
37
but the one inVersion_2B is obviously better practically.