Download Some previous powerpoint slides by Dr. Welch

CSCE 411
Design and Analysis of
Algorithms
Background Review
Prof. Jennifer Welch
CSCE 411, Background Review
1
Outline



Algorithm Analysis Tools
Sorting
Graph Algorithms
CSCE 411, Background Review
2
Algorithm Analysis Tools


Big-Oh notation
Basic probability
CSCE 411, Background Review
3
How to Measure Efficiency

Machine-independent way:


analyze "pseudocode" version of algorithm
assume idealized machine model


"Big-Oh" notation


one instruction takes one time unit
order of magnitude as problem size increases
Worst-case analyses

safe, often occurs most often, average case often
just as bad
CSCE 411, Background Review
4
Faster Algorithm vs. Faster CPU



A faster algorithm running on a slower machine will
always win for large enough instances
Suppose algorithm S1 sorts n keys in 2n2 instructions
Suppose computer C1 executes 1 billion instruc/sec



When n = 1 million, takes 2000 sec
Suppose algorithm S2 sorts n keys in 50nlog2n
instructions
Suppose computer C2 executes 10 million instruc/sec

When n = 1 million, takes 100 sec
CSCE 411, Background Review
5
Caveat


No point in finding fastest algorithm for part
of the program that is not the bottleneck
If program will only be run a few times, or
time is not an issue (e.g., run overnight), then
no point in finding fastest algorithm
CSCE 411, Background Review
6
Review of Big-Oh Notation

Measure time as a function of input size and ignore







multiplicative constants
lower order terms
f(n) = O(g(n)) : f is “≤” g
f(n) = Ω(g(n)) : f is “≥” g
f(n) = (g(n)) : f is “=” g
f(n) = o(g(n)) : f is “<” g
f(n) = (g(n)) : f is “>” g
CSCE 411, Background Review
7
Big-Oh



f(n) = O(g(n)) means, informally, that g is an
upper bound on f
Formally: there exist c > 0, n0 > 0 s.t.
f(n) ≤ cg(n) for all n ≥ n0.
To show that f(n) = O(g(n)), either find c and
n0 that satisfy the definition, or use this fact:

if limn->∞ f(n)/g(n) = some nonnegative constant,
then f(n) = O(g(n)).
CSCE 411, Background Review
8
Big-Omega



f(n) = Ω(g(n)) means, informally, that g is an
lower bound on f
Formally: there exist c > 0, n0 > 0 s.t.
f(n) ≥ cg(n) for all n ≥ n0.
To show that f(n) = Ω(g(n)), either find c and
n0 that satisfy the definition, or use this fact:

if limn->∞ f(n)/g(n) = some positive constant or ∞,
then f(n) = Ω(g(n)).
CSCE 411, Background Review
9
Big-Theta



f(n) = Θ(g(n)) means, informally, that g is a
tight bound on f
Formally: there exist c1 > 0, c2 > 0, n0 > 0 s.t.
c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0.
To show that f(n) = Θ(g(n)), either find c1, c2,
n0 that satisfy the definition, or use this fact:

if limn->∞ f(n)/g(n) = some positive constant,
then f(n) = Θ(g(n)).
CSCE 411, Background Review
10
Tying Back to Algorithm Analysis


Algorithm has worst case running time O(g(n))
if there exist c > 0 and n0 > 0 s.t. for all n ≥
n0, every execution (including the slowest) on
an input of size n takes at most cg(n) time.
Algorithm has worst case running time Ω(g(n))
if there exist c > 0 and n0 > 0 s.t. for all n ≥
n0, at least one execution (including the
slowest) on an input of size n takes at least
cg(n) time.
CSCE 411, Background Review
11
More Definitions

f(n) = o(g(n)) means, informally, g is a nontight upper bound on f



limn->∞ f(n)/g(n) = 0
pronounced “little-oh”
f(n) = ω(g(n)) means, informally, g is a nontight lower bound on f


limn->∞ f(n)/g(n) = ∞
pronounced “little-omega”
CSCE 411, Background Review
12
How To Solve Recurrences

Ad hoc method:




expand several times
guess the pattern
can verify with proof by induction
Master theorem

general formula that works if recurrence has the form
T(n) = aT(n/b) + f(n)



a is number of subproblems
n/b is size of each subproblem
f(n) is cost of non-recursive part
CSCE 411, Spring 2012: Set 3
13
Probability


Every probabilistic claim ultimately refers to some
sample space, which is a set of elementary events
Think of each elementary event as the outcome of
some experiment


Ex: flipping two coins gives sample space
{HH, HT, TH, TT}
An event is a subset of the sample space

Ex: event "both coins flipped the same" is {HH, TT}
CSCE 411, Spring 2012: Set 10
14
Sample Spaces and Events
HT
A
HH
TT
S
CSCE 411, Spring 2012: Set 10
TH
15
Probability Distribution

A probability distribution Pr on a sample
space S is a function from events of S to real
numbers s.t.




Pr[A] ≥ 0 for every event A
Pr[S] = 1
Pr[A U B] = Pr[A] + Pr[B] for every two nonintersecting ("mutually exclusive") events A and B
Pr[A] is the probability of event A
CSCE 411, Spring 2012: Set 10
16
Probability Distribution
Useful facts:
 Pr[Ø] = 0
 If A  B, then Pr[A] ≤ Pr[B]
 Pr[S — A] = 1 — Pr[A] // complement
 Pr[A U B] = Pr[A] + Pr[B] – Pr[A  B]
≤ Pr[A] + Pr[B]
CSCE 411, Spring 2012: Set 10
17
Probability Distribution
B
A
Pr[A U B] = Pr[A] + Pr[B] – Pr[A  B]
CSCE 411, Spring 2012: Set 10
18
Example
Suppose Pr[{HH}] = Pr[{HT}] = Pr[{TH}] = Pr[{TT}] = 1/4.
 Pr["at least one head"]
= Pr[{HH U HT U TH}]
= Pr[{HH}] + Pr[{HT}] + Pr[{TH}]
1/4
HT
= 3/4.
1/4
1/4
 Pr["less than one head"]
HH
TH
= 1 — Pr["at least one head"]
TT 1/4
= 1 — 3/4 = 1/4

CSCE 411, Spring 2012: Set 10
19
Specific Probability Distribution

discrete probability distribution: sample
space is finite or countably infinite


Ex: flipping two coins once; flipping one coin
infinitely often
uniform probability distribution: sample
space S is finite and every elementary event
has the same probability, 1/|S|

Ex: flipping two fair coins once
CSCE 411, Spring 2012: Set 10
20
Flipping a Fair Coin





Suppose we flip a fair coin n times
Each elementary event in the sample space is one
sequence of n heads and tails, describing the
outcome of one "experiment"
The size of the sample space is 2n.
Let A be the event "k heads and nk tails occur".
Pr[A] = C(n,k)/2n.

There are C(n,k) sequences of length n in which k heads and
n–k tails occur, and each has probability 1/2n.
CSCE 411, Spring 2012: Set 10
21
Example



n = 5, k = 3
Event A is
{HHHTT, HHTTH, HTTHH, TTHHH,
HHTHT, HTHTH, THTHH,
HTHHT, THHTH, THHHT}
Pr[3 heads and 2 tails] = C(5,3)/25
= 10/32
CSCE 411, Spring 2012: Set 10
22
Flipping Unfair Coins


Suppose we flip two coins, each of which
gives heads two-thirds of the time
What is the probability distribution on the
sample space?
4/9
HT
2/9
2/9
HH
TH
Pr[at least one head] = 8/9
TT 1/9
CSCE 411, Spring 2012: Set 10
23
Independent Events


Two events A and B are independent if
Pr[A  B] = Pr[A]·Pr[B]
I.e., probability that both A and B occur is the
product of the separate probabilities that A
occurs and that B occurs.
CSCE 411, Spring 2012: Set 10
24
Independent Events Example
In two-coin-flip example with fair coins:
 A = "first coin is heads"
 B = "coins are different"
A
1/4
HT
B
1/4
1/4
HH
TH
Pr[A] = 1/2
Pr[B] = 1/2
Pr[A  B] = 1/4 = (1/2)(1/2)
so A and B are independent
TT 1/4
CSCE 411, Spring 2012: Set 10
25
Discrete Random Variables




A discrete random variable X is a function from a finite
or countably infinite sample space to the real
numbers.
Associates a real number with each possible outcome
of an experiment
Define the event "X = v" to be the set of all the
elementary events s in the sample space with X(s) = v.
Pr["X = v"] is the sum of Pr[{s}] over all s with X(s) = v.
CSCE 411, Spring 2012: Set 10
26
Discrete Random Variable
X=v
X=v
X=v
X=v
X=v
Add up the probabilities of all the elementary events in
the orange event to get the probability that X = v
CSCE 411, Spring 2012: Set 10
27
Random Variable Example






Roll two fair 6-sided dice.
Sample space contains 36 elementary events
(1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 2:1,…)
Probability of each elementary event is 1/36
Define random variable X to be the maximum of
the two values rolled
What is Pr["X = 3"]?
It is 5/36, since there are 5 elementary events
with max value 3 (1:3, 2:3, 3:3, 3:2, and 3:1)
CSCE 411, Spring 2012: Set 10
28
Independent Random Variables

It is common for more than one random
variable to be defined on the same sample
space. E.g.:



X is maximum value rolled
Y is sum of the two values rolled
Two random variables X and Y are
independent if for all v and w, the events "X =
v" and "Y = w" are independent.
CSCE 411, Spring 2012: Set 10
29
Expected Value of a Random
Variable

Most common summary of a random variable
is its "average", weighted by the probabilities


called expected value, or expectation, or mean
Definition: E[X] = ∑ v Pr[X = v]
v
CSCE 411, Spring 2012: Set 10
30
Expected Value Example






Consider a game in which you flip two fair coins.
You get $3 for each head but lose $2 for each tail.
What are your expected earnings?
I.e., what is the expected value of the random
variable X, where X(HH) = 6, X(HT) = X(TH) = 1, and
X(TT) = —4?
Note that no value other than 6, 1, and —4 can be
taken on by X (e.g., Pr[X = 5] = 0).
E[X] = 6(1/4) + 1(1/4) + 1(1/4) + (—4)(1/4) = 1
CSCE 411, Spring 2012: Set 10
31
Properties of Expected Values



E[X+Y] = E[X] + E[Y], for any two random
variables X and Y, even if they are not
independent!
E[a·X] = a·E[X], for any random variable X and
any constant a.
E[X·Y] = E[X]·E[Y], for any two independent
random variables X and Y
CSCE 411, Spring 2012: Set 10
32
Conditional Probability


Formalizes having partial knowledge about the
outcome of an experiment
Example: flip two fair coins.



Probability of two heads is 1/4
Probability of two heads when you already know that the first
coin is a head is 1/2
Conditional probability of A given that B occurs,
denoted Pr[A|B], is defined to be
Pr[AB]/Pr[B]
CSCE 411, Spring 2012: Set 10
33
Conditional Probability
A
Pr[A] = 5/12
Pr[B] = 7/12
Pr[AB] = 2/12
Pr[A|B] = (2/12)/(7/12) = 2/7
B
CSCE 411, Spring 2012: Set 10
34
Conditional Probability

Definition is Pr[A|B] = Pr[AB]/Pr[B]

Equivalently, Pr[AB] = Pr[A|B]·Pr[B]
CSCE 411, Spring 2012: Set 10
35
Sorting




Insertion Sort
Heapsort
Mergesort
Quicksort
CSCE 411, Background Review
36
Insertion Sort Review

How it works:




incrementally build up longer and longer prefix of the array of
keys that is in sorted order
take the current key, find correct place in sorted prefix, and
shift to make room to insert it
Finding the correct place relies on comparing
current key to keys in sorted prefix
Worst-case running time is (n2)
CSCE 411, Spring 2012: Set 2
37
Heapsort Review

How it works:




put the keys in a heap data structure
repeatedly remove the min from the heap
Manipulating the heap involves comparing
keys to each other
Worst-case running time is (n log n)
CSCE 411, Spring 2012: Set 2
38
Mergesort Review

How it works:





split the array of keys in half
recursively sort the two halves
merge the two sorted halves
Merging the two sorted halves involves
comparing keys to each other
Worst-case running time is (n log n)
CSCE 411, Spring 2012: Set 2
39
Mergesort Example
5 2 4 6 1 3 2 6
5 2 4 6
5 2
5
4 6
2
2 5
1 3 2 6
4
1 3
6
4 6
2 4 5 6
1
2 6
3
1 3
2
6
2 6
1 2 3 6
1 2 2 3 4 5 6 6
CSCE 411, Spring 2012: Set 3
40
Recurrence Relation for
Mergesort



Let T(n) be worst case time on a sequence of
n keys
If n = 1, then T(n) = (1) (constant)
If n > 1, then T(n) = 2 T(n/2) + (n)



two subproblems of size n/2 each that are solved
recursively
(n) time to do the merge
Solving recurrence gives T(n) =  (n log n)
CSCE 411, Spring 2012: Set 3
41
Quicksort Review

How it works:





choose one key to be the pivot
partition the array of keys into those keys < the
pivot and those ≥ the pivot
recursively sort the two partitions
Partitioning the array involves comparing
keys to the pivot
Worst-case running time is (n2)
CSCE 411, Spring 2012: Set 2
42
Graphs


Mathematical definitions
Representations


Traversals


breadth-first search, depth-first search
Minimum spanning trees


adjacency list, adjacency matrix
Kruskal’s algorithm, Prim’s algorithm
Single-source shortest paths

Dijkstra’s algorithm, Bellman-Ford algorithm
CSCE 411, Background Review
43
Graphs

Directed graph G = (V,E)




In an undirected graph, edges are unordered pairs of
vertices
If (u,v) is in E, then v is adjacent to u and (u,v) is incident
from u and incident to v; v is neighbor of u


V is a finite set of vertices
E is the edge set, a binary relation on E (ordered pairs of vertice)
in an undirected graph, u is also adjacent to v, and (u,v) is incident
on u and v
Degree of a vertex is number of edges incident from or to
(or on) the vertex
CSCE 411, Spring 2012: Set 4
44
More on Graphs

A path of length k from vertex u to vertex u’ is
a sequence of k+1 vertices s.t. there is an
edge from each vertex to the next in the
sequence




In this case, u’ is reachable from u
A path is simple if no vertices are repeated
A path forms a cycle if last vertex = first
vertex
A cycle is simple if no vertices are repeated
other than first and last
CSCE 411, Spring 2012: Set 4
45
Graph Representations



So far, we have discussed graphs purely as
mathematical concepts
How can we represent a graph in a computer
program? We need some data structures.
Two most common representations are:


adjacency list – best for sparse graphs (few edges)
adjacency matrix – best for dense graphs (lots of
edges)
CSCE 411, Spring 2012: Set 4
46
Adjacency List Representation



Given graph G = (V,E)
array Adj of |V| linked lists, one for each vertex in V
The adjacency list for vertex u, Adj[u], contains all
vertices v s.t. (u,v) is in E



each vertex in the list might be just a string, representing its
name, or it might be some other data structure
representing the vertex, or it might be a pointer to the data
structure for that vertex
Requires Θ(V+E) space (memory)
Requires Θ(degree(u)) time to check if (u,v) is in E
CSCE 411, Spring 2012: Set 4
47
Adjacency List Representation
a
b
e
c
d
a
b
c
b
c
a
a
d
d
e
d
e
b
b
c
d
e
Space-efficient for sparse graphs
CSCE 411, Spring 2012: Set 8
48
Adjacency Matrix Representation



Given graph G = (V,E)
Number the vertices 1, 2, ..., |V|
Use a |V| x |V| matrix A with (i,j)-entry of A being




1 if (i,j) is in E
0 if (i,j) is not in E
Requires Θ(V2) space (memory)
Requires Θ(1) time to check if (u,v) is in E
CSCE 411, Spring 2012: Set 4
49
Adjacency Matrix Representation
a
b
e
c
d
a b c
a 0 1 1
b 1 0 0
c 1 0 0
d 0 1 1
e 0 1 0
d
0
1
1
0
1
e
0
1
0
1
0
Check for an edge in constant time
CSCE 411, Spring 2012: Set 8
50
More on Graph Representations

Read Ch 22, Sec 1 for



more about directed vs. undirected graphs
how to represented weighted graphs (some value is
associated with each edge)
pros and cons of adjacency list vs. adjacency matrix
CSCE 411, Spring 2012: Set 4
51
Graph Traversals

Ways to traverse/search a graph



Visit every vertex exactly once
Breadth-First Search
Depth-First Search
CSCE 411, Spring 2012: Set 8
52
Breadth First Search (BFS)


Input: G = (V,E) and source s in V
for each vertex v in V do




mark v as unvisited
mark s as visited
enq(Q,s) // FIFO queue Q
while Q is not empty do


u := deq(Q)
for each unvisited neighbor v of u do


mark v as visited
enq(Q,v)
CSCE 411, Spring 2012: Set 8
53
BFS Example

start at s and consider in alphabetical order
a
b
s
c
d
CSCE 411, Spring 2012: Set 8
54
BFS Tree

We can make a spanning tree rooted at s by
remembering the "parent" of each vertex
CSCE 411, Spring 2012: Set 8
55
Breadth First Search #2





Input: G = (V,E) and source s in V
for each vertex v in V do

mark v as unvisited

parent[v] := nil
mark s as visited
parent[s] := s
enq(Q,s) // FIFO queue Q
CSCE 411, Spring 2012: Set 8
56
Breadth First Search #2

while Q is not empty do


u := deq(Q)
for each unvisited neighbor v of u do



mark v as visited
parent[v] := u
enq(Q,v)
CSCE 411, Spring 2012: Set 8
57
BFS Tree Example
a
b
s
c
d
CSCE 411, Spring 2012: Set 8
58
BFS Trees


BFS tree is not necessarily unique for a given
graph
Depends on the order in which neighboring
vertices are processed
CSCE 411, Spring 2012: Set 8
59
BFS Numbering


During the breadth-first search, assign an
integer to each vertex
Indicate the distance of each vertex from the
source s
CSCE 411, Spring 2012: Set 8
60
Breadth First Search #3


Input: G = (V,E) and source s in V
for each vertex v in V do







mark v as unvisited
parent[v] := nil
d[v] := infinity
mark s as visited
parent[s] := s
d[s] := 0
enq(Q,s) // FIFO queue Q
CSCE 411, Spring 2012: Set 8
61
Breadth First Search #3

while Q is not empty do


u := deq(Q)
for each unvisited neighbor v of u do




mark v as visited
parent[v] := u
d[v] := d[u] + 1
enq(Q,v)
CSCE 411, Spring 2012: Set 8
62
BFS Numbering Example
d=1
a
d=0
b d=2
s
c d=2
d
d=1
CSCE 411, Spring 2012: Set 8
63
Shortest Path Tree

Theorem: BFS algorithm



visits all and only vertices reachable from s
sets d[v] equal to the shortest path distance from s
to v, for all vertices v, and
sets parent variables to form a shortest path tree
CSCE 411, Spring 2012: Set 8
64
BFS Running Time




Initialization of each vertex takes O(V) time
Every vertex is enqueued once and dequeued once,
taking O(V) time
When a vertex is dequeued, all its neighbors are
checked to see if they are unvisited, taking time
proportional to number of neighbors of the vertex,
and summing to O(E) over all iterations
Total time is O(V+E)
CSCE 411, Spring 2012: Set 8
65
Depth-First Search


Input: G = (V,E)
for each vertex u in V do


for each unvisited vertex u in V do


mark u as unvisited
call recursiveDFS(u)
recursiveDFS(u):


mark u as visited
for each unvisited neighbor v of u do

call recursiveDFS(v)
CSCE 411, Spring 2012: Set 8
66
DFS Example

start at a and consider in alphabetical order
a
b
c
d
e
f
g
h
CSCE 411, Spring 2012: Set 8
67
Disconnected Graphs


What if graph is disconnected or is directed?
call DFS on several vertices to visit all vertices

purpose of second for-loop in non-recursive
wrapper
a
b
d
c
e
CSCE 411, Spring 2012: Set 8
68
DFS Tree


Actually might be a DFS forest (collection of
trees)
Keep track of parents
CSCE 411, Spring 2012: Set 8
69
Depth-First Search #2


Input: G = (V,E)
for each vertex u in V do



mark u as unvisited
parent[u] := nil

recursiveDFS(u):


for each unvisited vertex
u in V do


parent[u] := u // a root
call recursive DFS(u)
CSCE 411, Spring 2012: Set 8
mark u as visited
for each unvisited
neighbor v of u do


parent[v] := u
call recursiveDFS(v)
70
More Properties of DFS



Need to keep track of more information for
each vertex:
discovery time: when its recursive call starts
finish time: when its recursive call ends
CSCE 411, Spring 2012: Set 8
71
Depth-First Search #3


Input: G = (V,E)
for each vertex u in V do




mark u as unvisited
parent[u] := nil

recursiveDFS(u):


time := 0
for each unvisited vertex u
in V do


parent[u] := u // a root
call recursive DFS(u)


mark u as visited
time++
disc[u] := time
for each unvisited
neighbor v of u do




CSCE 411, Spring 2012: Set 8
parent[v] := u
call recursiveDFS(v)
time++
fin[u] := time
72
Running Time of DFS





initialization takes O(V) time
second for loop in non-recursive wrapper
considers each vertex, so O(V) iterations
one recursive call is made for each vertex
in recursive call for vertex u, all its neighbors
are checked; total time in all recursive calls is
O(E)
Total time is O(V+E)
CSCE 411, Spring 2012: Set 8
73
Minimum Spanning Tree
16
5
12
4
7
3
6
11
14
9
8
10
2
15
13
17
18
Given a connected undirected graph with edge weights,
find subset of edges that spans all the nodes,
creates no cycle, and minimizes sum of weights
CSCE 411, Spring 2012: Set 4
74
Facts About MSTs



There can be many spanning trees of a graph
In fact, there can be many minimum
spanning trees of a graph
But if every edge has a unique weight, then
there is a unique MST
CSCE 411, Spring 2012: Set 4
75
Generic MST Algorithm



input: weighted undirected graph
G = (V,E,w)
T := empty set
while T is not yet a spanning tree of G



find an edge e in E s.t. T U {e} is a subgraph of some
MST of G
add e to T
return T (as MST of G)
CSCE 411, Spring 2012: Set 9
76
Kruskal's MST algorithm
16
5
12
4
7
3
6
11
14
9
8
10
2
15
13
17
18
consider the edges in increasing order of weight,
add in an edge iff it does not cause a cycle
CSCE 411, Spring 2012: Set 4
77
Implementing Kruskal's Alg.

Sort edges by weight


efficient algorithms known
How to test quickly if adding in the next edge
would cause a cycle?

use disjoint set data structure, later
CSCE 411, Spring 2012: Set 4
78
Kruskal's Algorithm as a Special
Case of Generic Algorithm



Consider edges in increasing order of weight
Add the next edge iff it doesn't cause a cycle
At any point, T is a forest (set of trees);
eventually T is a single tree
CSCE 411, Spring 2012: Set 9
79
Another MST Algorithm


Kruskal's algorithm maintains a forest that
grows until it forms a spanning tree
Alternative idea is keep just one tree and
grow it until it spans all the nodes


Prim's algorithm
At each iteration, choose the minimum weight
outgoing edge to add

greedy!
CSCE 411, Spring 2012: Set 4
80
Idea of Prim's Algorithm


Instead of growing the MST as possibly
multiple trees that eventually all merge, grow
the MST from a single vertex, so that there is
only one tree at any point.
Also a special case of the generic algorithm:
at each step, add the minimum weight edge
that goes out from the tree constructed so
far.
CSCE 411, Spring 2012: Set 9
81
Prim's Algorithm




input: weighted undirected graph G = (V,E,w)
T := empty set
S := {any vertex in V}
while |T| < |V| - 1 do




let (u,v) be a min wt. outgoing edge (u in S, v not in S)
add (u,v) to T
add v to S
return (S,T) (as MST of G)
CSCE 411, Spring 2012: Set 9
82
Prim's Algorithm Example
8
b
4
c
d
9
2
11
a
7
i
4
7
8
h
6
1
g
2
CSCE 411, Spring 2012: Set 9
e
14
f
10
83
Implementing Prim's Algorithm




How do we find minimum weight outgoing
edge?
First cut: scan all adjacency lists at each
iteration.
Results in O(VE) time.
Try to do better.
CSCE 411, Spring 2012: Set 9
84
Implementing Prim's Algorithm


Idea: have each vertex not yet in the tree
keep track of its best (cheapest) edge to the
tree constructed so far.
To find min wt. outgoing edge, find minimum
among these values

use a priority queue to store the best edge info
(insert and extract-min operations)
CSCE 411, Spring 2012: Set 9
85
Implementing Prim's Algorithm

When a vertex v is added to T, some other
vertices might have their best edges affected,
but only neighbors of v

add decrease-key operation to the priority queue
v's best edge to Ti
x
Ti
u
Ti+1
v
check if this edge is cheaper
for w
w
x's best edge to Ti
w's best edge to Ti
CSCE 411, Spring 2012: Set 9
86
Details on Prim's Algorithm
Associate with each vertex v two fields:
 best-wt[v] : if v is not yet in the tree, then it
holds the min. wt. of all edges from v to a
vertex in the tree. Initially infinity.
 best-node[v] : if v is not yet in the tree, then it
holds the name of the vertex (node) u in the
tree s.t. w(v,u) is v's best-wt. Initially nil.
CSCE 411, Spring 2012: Set 9
87
Details on Prim's Algorithm
input: G = (V,E,w)
// initialization
 initialize priority queue Q to contain all vertices,
using best-wt values as keys
 let v0 be any vertex in V
 decrease-key(Q,v0,0)
// last line means change best-wt[v0] to 0 and adjust Q
accordingly

CSCE 411, Spring 2012: Set 9
88
Details on Prim's Algorithm

while Q is not empty do



u := extract-min(Q) // vertex w/ smallest best-wt
if u is not v0 then add (u,best-node[u]) to T
for each neighbor v of u do

if v is in Q and w(u,v) < best-wt[v] then
 best-node[v] := u
 decrease-key(Q,v,w(u,v))

return (V,T) // as MST of G
CSCE 411, Spring 2012: Set 9
89
Running Time of Prim's Algorithm
Depends on priority queue implementation. Let
 Tins be time for insert
 Tdec be time for decrease-key
 Tex be time for extract-min
Then we have
 |V| inserts and one decrease-key in the
initialization: O(VTins+Tdec)
 |V| iterations of while

one extract-min per iteration: O(VTex) total
CSCE 411, Spring 2012: Set 9
90
Running Time of Prim's Algorithm




Each iteration of while includes a for loop.
Number of iterations of for loop varies,
depending on how many neighbors the
current vertex has
Total number of iterations of for loop is O(E).
Each iteration of for loop:

one decrease key, so O(ETdec) total
CSCE 411, Spring 2012: Set 9
91
Running Time of Prim's Algorithm


O(V(Tins + Tex) + ETdec)
If priority queue is implemented with a binary
heap, then



Tins = Tex = Tdec = O(log V)
total time is O(E log V)
(Think about how to implement decrease-key
in O(log V) time.)
CSCE 411, Spring 2012: Set 9
92
Shortest Paths in a Graph

Let's review two important single-source
shortest path algorithms:


Dijkstra's algorithm
Bellman-Ford algorithm
CSCE 411, Spring 2012: Set 9
93
Single Source Shortest Path
Problem



Given: directed or undirected graph G =
(V,E,w) and source vertex s in V
Find: For each t in V, a path in G from s to t
with minimum weight
Warning! Negative weights are a problem:
s
4
t
5
CSCE 411, Spring 2012: Set 9
94
Shortest Path Tree



Result of a SSSP algorithm can be viewed as
a tree rooted at the source
Why not use breadth-first search?
Works fine if all weights are the same:


weight of each path is (a multiple of) the number of
edges in the path
Doesn't work when weights are different
CSCE 411, Spring 2012: Set 9
95
Dijkstra's SSSP Algorithm





Assumes all edge weights are nonnegative
Similar to Prim's MST algorithm
Start with source vertex s and iteratively construct a
tree rooted at s
Each vertex keeps track of tree vertex that provides
cheapest path from s (not just cheapest path from
any tree vertex)
At each iteration, include the vertex whose cheapest
path from s is the overall cheapest
CSCE 411, Spring 2012: Set 9
96
Prim's vs. Dijkstra's
4
5
4
1
s
5
s
6
Prim's MST
1
6
Dijkstra's SSSP
CSCE 411, Spring 2012: Set 9
97
Implementing Dijkstra's Alg.




How can each vertex u keep track of its best path
from s?
Keep an estimate, d[u], of shortest path distance
from s to u
Use d as a key in a priority queue
When u is added to the tree, check each of u's
neighbors v to see if u provides v with a cheaper
path from s:

compare d[v] to d[u] + w(u,v)
CSCE 411, Spring 2012: Set 9
98
Dijkstra's Algorithm
input: G = (V,E,w) and source vertex s
// initialization
 d[s] := 0
 d[v] := infinity for all other vertices v
 initialize priority queue Q to contain all
vertices using d values as keys

CSCE 411, Spring 2012: Set 9
99
Dijkstra's Algorithm

while Q is not empty do
 u := extract-min(Q)
 for each neighbor v of u do
 if d[u] + w(u,v) < d[v] then
 d[v] := d[u] + w(u,v)
 decrease-key(Q,v,d[v])
 parent(v) := u
CSCE 411, Spring 2012: Set 9
100
Dijkstra's Algorithm Example
a
2
iteration
b
12 8
10 6 c 3 4
d
2
4
e
source is vertex a
0
1
2
3 4 5
Q
abcde bcde
cde
de d
Ø
d[a]
0
0
0
0
0
0
d[b]
∞
2
2
2
2
2
d[c]
∞
12
10
10 10 10
d[d]
∞
∞
∞
16 13 13
d[e]
∞
∞
11
11 11 11
9
CSCE 411, Spring 2012: Set 9
101
Running Time of Dijkstra's Alg.

initialization: insert each vertex once


O(V) iterations of while loop



one extract-min per iteration => O(V Tex)
for loop inside while loop has variable number of iterations…
For loop has O(E) iterations total


O(V Tins)
one decrease-key per iteration => O(E Tdec)
Total is O(V (Tins + Tex) + E Tdec)
CSCE 411, Spring 2012: Set 9
102
Using Different Heap
Implementations


O(V(Tins + Tex) + ETdec)
If priority queue is implemented with a binary heap,
then



Tins = Tex = Tdec = O(log V)
total time is O(E log V)
There are fancier implementations of the priority
queue, such as Fibonacci heap:


Tins = O(1), Tex = O(log V), Tdec = O(1) (amortized)
total time is O(V log V + E)
CSCE 411, Spring 2012: Set 9
103
Using Simpler Heap
Implementations




O(V(Tins + Tex) + ETdec)
If graph is dense, so that |E| = (V2), then it doesn't
help to make Tins and Tex to be at most O(V).
Instead, focus on making Tdec be small, say
constant.
Implement priority queue with an unsorted array:


Tins = O(1), Tex = O(V), Tdec = O(1)
total is O(V2)
CSCE 411, Spring 2012: Set 9
104
What About Negative Edge
Weights?

Dijkstra's SSSP algorithm requires all edge
weights to be nonnegative


even more restrictive than outlawing negative
weight cycles
Bellman-Ford SSSP algorithm can handle
negative edge weights

even "handles" negative weight cycles by reporting
they exist
CSCE 411, Spring 2012: Set 9
105
Bellman-Ford Idea

Consider each edge (u,v) and see if u offers v
a cheaper path from s


compare d[v] to d[u] + w(u,v)
Repeat this process |V| - 1 times to ensure
that accurate information propgates from s,
no matter what order the edges are
considered in
CSCE 411, Spring 2012: Set 9
106
Bellman-Ford SSSP Algorithm
input: directed or undirected graph G = (V,E,w)
//initialization
 initialize d[v] to infinity and parent[v] to nil for all v in V other than the
source
 initialize d[s] to 0 and parent[s] to s
// main body
 for i := 1 to |V| - 1 do
 for each (u,v) in E do
// consider in arbitrary order
 if d[u] + w(u,v) < d[v] then
 d[v] := d[u] + w(u,v)
 parent[v] := u

CSCE 411, Spring 2012: Set 9
107
Bellman-Ford SSSP Algorithm
// check for negative weight cycles
 for each (u,v) in E do
 if d[u] + w(u,v) < d[v] then
 output "negative weight cycle exists"
CSCE 411, Spring 2012: Set 9
108
Running Time of Bellman-Ford



O(V) iterations of outer for loop
O(E) iterations of inner for loop
O(VE) time total
CSCE 411, Spring 2012: Set 9
109
Bellman-Ford Example
a
3
s
2
b
—4
4
c
1
process edges in order
(c,b)
(a,b)
(c,a)
(s,a)
(s,c)
<board work>
CSCE 411, Spring 2012: Set 9
110
Speeding Up Bellman-Ford

The previous example would have converged
faster if we had considered the edges in a
different order in the for loop


move outward from s
If the graph is a DAG (no cycles), we can fully
exploit this idea to speed up the running time
CSCE 411, Spring 2012: Set 9
111