Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
CSCE 411 Design and Analysis of Algorithms Background Review Prof. Jennifer Welch CSCE 411, Background Review 1 Outline Algorithm Analysis Tools Sorting Graph Algorithms CSCE 411, Background Review 2 Algorithm Analysis Tools Big-Oh notation Basic probability CSCE 411, Background Review 3 How to Measure Efficiency Machine-independent way: analyze "pseudocode" version of algorithm assume idealized machine model "Big-Oh" notation one instruction takes one time unit order of magnitude as problem size increases Worst-case analyses safe, often occurs most often, average case often just as bad CSCE 411, Background Review 4 Faster Algorithm vs. Faster CPU A faster algorithm running on a slower machine will always win for large enough instances Suppose algorithm S1 sorts n keys in 2n2 instructions Suppose computer C1 executes 1 billion instruc/sec When n = 1 million, takes 2000 sec Suppose algorithm S2 sorts n keys in 50nlog2n instructions Suppose computer C2 executes 10 million instruc/sec When n = 1 million, takes 100 sec CSCE 411, Background Review 5 Caveat No point in finding fastest algorithm for part of the program that is not the bottleneck If program will only be run a few times, or time is not an issue (e.g., run overnight), then no point in finding fastest algorithm CSCE 411, Background Review 6 Review of Big-Oh Notation Measure time as a function of input size and ignore multiplicative constants lower order terms f(n) = O(g(n)) : f is “≤” g f(n) = Ω(g(n)) : f is “≥” g f(n) = (g(n)) : f is “=” g f(n) = o(g(n)) : f is “<” g f(n) = (g(n)) : f is “>” g CSCE 411, Background Review 7 Big-Oh f(n) = O(g(n)) means, informally, that g is an upper bound on f Formally: there exist c > 0, n0 > 0 s.t. f(n) ≤ cg(n) for all n ≥ n0. To show that f(n) = O(g(n)), either find c and n0 that satisfy the definition, or use this fact: if limn->∞ f(n)/g(n) = some nonnegative constant, then f(n) = O(g(n)). CSCE 411, Background Review 8 Big-Omega f(n) = Ω(g(n)) means, informally, that g is an lower bound on f Formally: there exist c > 0, n0 > 0 s.t. f(n) ≥ cg(n) for all n ≥ n0. To show that f(n) = Ω(g(n)), either find c and n0 that satisfy the definition, or use this fact: if limn->∞ f(n)/g(n) = some positive constant or ∞, then f(n) = Ω(g(n)). CSCE 411, Background Review 9 Big-Theta f(n) = Θ(g(n)) means, informally, that g is a tight bound on f Formally: there exist c1 > 0, c2 > 0, n0 > 0 s.t. c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0. To show that f(n) = Θ(g(n)), either find c1, c2, n0 that satisfy the definition, or use this fact: if limn->∞ f(n)/g(n) = some positive constant, then f(n) = Θ(g(n)). CSCE 411, Background Review 10 Tying Back to Algorithm Analysis Algorithm has worst case running time O(g(n)) if there exist c > 0 and n0 > 0 s.t. for all n ≥ n0, every execution (including the slowest) on an input of size n takes at most cg(n) time. Algorithm has worst case running time Ω(g(n)) if there exist c > 0 and n0 > 0 s.t. for all n ≥ n0, at least one execution (including the slowest) on an input of size n takes at least cg(n) time. CSCE 411, Background Review 11 More Definitions f(n) = o(g(n)) means, informally, g is a nontight upper bound on f limn->∞ f(n)/g(n) = 0 pronounced “little-oh” f(n) = ω(g(n)) means, informally, g is a nontight lower bound on f limn->∞ f(n)/g(n) = ∞ pronounced “little-omega” CSCE 411, Background Review 12 How To Solve Recurrences Ad hoc method: expand several times guess the pattern can verify with proof by induction Master theorem general formula that works if recurrence has the form T(n) = aT(n/b) + f(n) a is number of subproblems n/b is size of each subproblem f(n) is cost of non-recursive part CSCE 411, Spring 2012: Set 3 13 Probability Every probabilistic claim ultimately refers to some sample space, which is a set of elementary events Think of each elementary event as the outcome of some experiment Ex: flipping two coins gives sample space {HH, HT, TH, TT} An event is a subset of the sample space Ex: event "both coins flipped the same" is {HH, TT} CSCE 411, Spring 2012: Set 10 14 Sample Spaces and Events HT A HH TT S CSCE 411, Spring 2012: Set 10 TH 15 Probability Distribution A probability distribution Pr on a sample space S is a function from events of S to real numbers s.t. Pr[A] ≥ 0 for every event A Pr[S] = 1 Pr[A U B] = Pr[A] + Pr[B] for every two nonintersecting ("mutually exclusive") events A and B Pr[A] is the probability of event A CSCE 411, Spring 2012: Set 10 16 Probability Distribution Useful facts: Pr[Ø] = 0 If A B, then Pr[A] ≤ Pr[B] Pr[S — A] = 1 — Pr[A] // complement Pr[A U B] = Pr[A] + Pr[B] – Pr[A B] ≤ Pr[A] + Pr[B] CSCE 411, Spring 2012: Set 10 17 Probability Distribution B A Pr[A U B] = Pr[A] + Pr[B] – Pr[A B] CSCE 411, Spring 2012: Set 10 18 Example Suppose Pr[{HH}] = Pr[{HT}] = Pr[{TH}] = Pr[{TT}] = 1/4. Pr["at least one head"] = Pr[{HH U HT U TH}] = Pr[{HH}] + Pr[{HT}] + Pr[{TH}] 1/4 HT = 3/4. 1/4 1/4 Pr["less than one head"] HH TH = 1 — Pr["at least one head"] TT 1/4 = 1 — 3/4 = 1/4 CSCE 411, Spring 2012: Set 10 19 Specific Probability Distribution discrete probability distribution: sample space is finite or countably infinite Ex: flipping two coins once; flipping one coin infinitely often uniform probability distribution: sample space S is finite and every elementary event has the same probability, 1/|S| Ex: flipping two fair coins once CSCE 411, Spring 2012: Set 10 20 Flipping a Fair Coin Suppose we flip a fair coin n times Each elementary event in the sample space is one sequence of n heads and tails, describing the outcome of one "experiment" The size of the sample space is 2n. Let A be the event "k heads and nk tails occur". Pr[A] = C(n,k)/2n. There are C(n,k) sequences of length n in which k heads and n–k tails occur, and each has probability 1/2n. CSCE 411, Spring 2012: Set 10 21 Example n = 5, k = 3 Event A is {HHHTT, HHTTH, HTTHH, TTHHH, HHTHT, HTHTH, THTHH, HTHHT, THHTH, THHHT} Pr[3 heads and 2 tails] = C(5,3)/25 = 10/32 CSCE 411, Spring 2012: Set 10 22 Flipping Unfair Coins Suppose we flip two coins, each of which gives heads two-thirds of the time What is the probability distribution on the sample space? 4/9 HT 2/9 2/9 HH TH Pr[at least one head] = 8/9 TT 1/9 CSCE 411, Spring 2012: Set 10 23 Independent Events Two events A and B are independent if Pr[A B] = Pr[A]·Pr[B] I.e., probability that both A and B occur is the product of the separate probabilities that A occurs and that B occurs. CSCE 411, Spring 2012: Set 10 24 Independent Events Example In two-coin-flip example with fair coins: A = "first coin is heads" B = "coins are different" A 1/4 HT B 1/4 1/4 HH TH Pr[A] = 1/2 Pr[B] = 1/2 Pr[A B] = 1/4 = (1/2)(1/2) so A and B are independent TT 1/4 CSCE 411, Spring 2012: Set 10 25 Discrete Random Variables A discrete random variable X is a function from a finite or countably infinite sample space to the real numbers. Associates a real number with each possible outcome of an experiment Define the event "X = v" to be the set of all the elementary events s in the sample space with X(s) = v. Pr["X = v"] is the sum of Pr[{s}] over all s with X(s) = v. CSCE 411, Spring 2012: Set 10 26 Discrete Random Variable X=v X=v X=v X=v X=v Add up the probabilities of all the elementary events in the orange event to get the probability that X = v CSCE 411, Spring 2012: Set 10 27 Random Variable Example Roll two fair 6-sided dice. Sample space contains 36 elementary events (1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 2:1,…) Probability of each elementary event is 1/36 Define random variable X to be the maximum of the two values rolled What is Pr["X = 3"]? It is 5/36, since there are 5 elementary events with max value 3 (1:3, 2:3, 3:3, 3:2, and 3:1) CSCE 411, Spring 2012: Set 10 28 Independent Random Variables It is common for more than one random variable to be defined on the same sample space. E.g.: X is maximum value rolled Y is sum of the two values rolled Two random variables X and Y are independent if for all v and w, the events "X = v" and "Y = w" are independent. CSCE 411, Spring 2012: Set 10 29 Expected Value of a Random Variable Most common summary of a random variable is its "average", weighted by the probabilities called expected value, or expectation, or mean Definition: E[X] = ∑ v Pr[X = v] v CSCE 411, Spring 2012: Set 10 30 Expected Value Example Consider a game in which you flip two fair coins. You get $3 for each head but lose $2 for each tail. What are your expected earnings? I.e., what is the expected value of the random variable X, where X(HH) = 6, X(HT) = X(TH) = 1, and X(TT) = —4? Note that no value other than 6, 1, and —4 can be taken on by X (e.g., Pr[X = 5] = 0). E[X] = 6(1/4) + 1(1/4) + 1(1/4) + (—4)(1/4) = 1 CSCE 411, Spring 2012: Set 10 31 Properties of Expected Values E[X+Y] = E[X] + E[Y], for any two random variables X and Y, even if they are not independent! E[a·X] = a·E[X], for any random variable X and any constant a. E[X·Y] = E[X]·E[Y], for any two independent random variables X and Y CSCE 411, Spring 2012: Set 10 32 Conditional Probability Formalizes having partial knowledge about the outcome of an experiment Example: flip two fair coins. Probability of two heads is 1/4 Probability of two heads when you already know that the first coin is a head is 1/2 Conditional probability of A given that B occurs, denoted Pr[A|B], is defined to be Pr[AB]/Pr[B] CSCE 411, Spring 2012: Set 10 33 Conditional Probability A Pr[A] = 5/12 Pr[B] = 7/12 Pr[AB] = 2/12 Pr[A|B] = (2/12)/(7/12) = 2/7 B CSCE 411, Spring 2012: Set 10 34 Conditional Probability Definition is Pr[A|B] = Pr[AB]/Pr[B] Equivalently, Pr[AB] = Pr[A|B]·Pr[B] CSCE 411, Spring 2012: Set 10 35 Sorting Insertion Sort Heapsort Mergesort Quicksort CSCE 411, Background Review 36 Insertion Sort Review How it works: incrementally build up longer and longer prefix of the array of keys that is in sorted order take the current key, find correct place in sorted prefix, and shift to make room to insert it Finding the correct place relies on comparing current key to keys in sorted prefix Worst-case running time is (n2) CSCE 411, Spring 2012: Set 2 37 Heapsort Review How it works: put the keys in a heap data structure repeatedly remove the min from the heap Manipulating the heap involves comparing keys to each other Worst-case running time is (n log n) CSCE 411, Spring 2012: Set 2 38 Mergesort Review How it works: split the array of keys in half recursively sort the two halves merge the two sorted halves Merging the two sorted halves involves comparing keys to each other Worst-case running time is (n log n) CSCE 411, Spring 2012: Set 2 39 Mergesort Example 5 2 4 6 1 3 2 6 5 2 4 6 5 2 5 4 6 2 2 5 1 3 2 6 4 1 3 6 4 6 2 4 5 6 1 2 6 3 1 3 2 6 2 6 1 2 3 6 1 2 2 3 4 5 6 6 CSCE 411, Spring 2012: Set 3 40 Recurrence Relation for Mergesort Let T(n) be worst case time on a sequence of n keys If n = 1, then T(n) = (1) (constant) If n > 1, then T(n) = 2 T(n/2) + (n) two subproblems of size n/2 each that are solved recursively (n) time to do the merge Solving recurrence gives T(n) = (n log n) CSCE 411, Spring 2012: Set 3 41 Quicksort Review How it works: choose one key to be the pivot partition the array of keys into those keys < the pivot and those ≥ the pivot recursively sort the two partitions Partitioning the array involves comparing keys to the pivot Worst-case running time is (n2) CSCE 411, Spring 2012: Set 2 42 Graphs Mathematical definitions Representations Traversals breadth-first search, depth-first search Minimum spanning trees adjacency list, adjacency matrix Kruskal’s algorithm, Prim’s algorithm Single-source shortest paths Dijkstra’s algorithm, Bellman-Ford algorithm CSCE 411, Background Review 43 Graphs Directed graph G = (V,E) In an undirected graph, edges are unordered pairs of vertices If (u,v) is in E, then v is adjacent to u and (u,v) is incident from u and incident to v; v is neighbor of u V is a finite set of vertices E is the edge set, a binary relation on E (ordered pairs of vertice) in an undirected graph, u is also adjacent to v, and (u,v) is incident on u and v Degree of a vertex is number of edges incident from or to (or on) the vertex CSCE 411, Spring 2012: Set 4 44 More on Graphs A path of length k from vertex u to vertex u’ is a sequence of k+1 vertices s.t. there is an edge from each vertex to the next in the sequence In this case, u’ is reachable from u A path is simple if no vertices are repeated A path forms a cycle if last vertex = first vertex A cycle is simple if no vertices are repeated other than first and last CSCE 411, Spring 2012: Set 4 45 Graph Representations So far, we have discussed graphs purely as mathematical concepts How can we represent a graph in a computer program? We need some data structures. Two most common representations are: adjacency list – best for sparse graphs (few edges) adjacency matrix – best for dense graphs (lots of edges) CSCE 411, Spring 2012: Set 4 46 Adjacency List Representation Given graph G = (V,E) array Adj of |V| linked lists, one for each vertex in V The adjacency list for vertex u, Adj[u], contains all vertices v s.t. (u,v) is in E each vertex in the list might be just a string, representing its name, or it might be some other data structure representing the vertex, or it might be a pointer to the data structure for that vertex Requires Θ(V+E) space (memory) Requires Θ(degree(u)) time to check if (u,v) is in E CSCE 411, Spring 2012: Set 4 47 Adjacency List Representation a b e c d a b c b c a a d d e d e b b c d e Space-efficient for sparse graphs CSCE 411, Spring 2012: Set 8 48 Adjacency Matrix Representation Given graph G = (V,E) Number the vertices 1, 2, ..., |V| Use a |V| x |V| matrix A with (i,j)-entry of A being 1 if (i,j) is in E 0 if (i,j) is not in E Requires Θ(V2) space (memory) Requires Θ(1) time to check if (u,v) is in E CSCE 411, Spring 2012: Set 4 49 Adjacency Matrix Representation a b e c d a b c a 0 1 1 b 1 0 0 c 1 0 0 d 0 1 1 e 0 1 0 d 0 1 1 0 1 e 0 1 0 1 0 Check for an edge in constant time CSCE 411, Spring 2012: Set 8 50 More on Graph Representations Read Ch 22, Sec 1 for more about directed vs. undirected graphs how to represented weighted graphs (some value is associated with each edge) pros and cons of adjacency list vs. adjacency matrix CSCE 411, Spring 2012: Set 4 51 Graph Traversals Ways to traverse/search a graph Visit every vertex exactly once Breadth-First Search Depth-First Search CSCE 411, Spring 2012: Set 8 52 Breadth First Search (BFS) Input: G = (V,E) and source s in V for each vertex v in V do mark v as unvisited mark s as visited enq(Q,s) // FIFO queue Q while Q is not empty do u := deq(Q) for each unvisited neighbor v of u do mark v as visited enq(Q,v) CSCE 411, Spring 2012: Set 8 53 BFS Example start at s and consider in alphabetical order a b s c d CSCE 411, Spring 2012: Set 8 54 BFS Tree We can make a spanning tree rooted at s by remembering the "parent" of each vertex CSCE 411, Spring 2012: Set 8 55 Breadth First Search #2 Input: G = (V,E) and source s in V for each vertex v in V do mark v as unvisited parent[v] := nil mark s as visited parent[s] := s enq(Q,s) // FIFO queue Q CSCE 411, Spring 2012: Set 8 56 Breadth First Search #2 while Q is not empty do u := deq(Q) for each unvisited neighbor v of u do mark v as visited parent[v] := u enq(Q,v) CSCE 411, Spring 2012: Set 8 57 BFS Tree Example a b s c d CSCE 411, Spring 2012: Set 8 58 BFS Trees BFS tree is not necessarily unique for a given graph Depends on the order in which neighboring vertices are processed CSCE 411, Spring 2012: Set 8 59 BFS Numbering During the breadth-first search, assign an integer to each vertex Indicate the distance of each vertex from the source s CSCE 411, Spring 2012: Set 8 60 Breadth First Search #3 Input: G = (V,E) and source s in V for each vertex v in V do mark v as unvisited parent[v] := nil d[v] := infinity mark s as visited parent[s] := s d[s] := 0 enq(Q,s) // FIFO queue Q CSCE 411, Spring 2012: Set 8 61 Breadth First Search #3 while Q is not empty do u := deq(Q) for each unvisited neighbor v of u do mark v as visited parent[v] := u d[v] := d[u] + 1 enq(Q,v) CSCE 411, Spring 2012: Set 8 62 BFS Numbering Example d=1 a d=0 b d=2 s c d=2 d d=1 CSCE 411, Spring 2012: Set 8 63 Shortest Path Tree Theorem: BFS algorithm visits all and only vertices reachable from s sets d[v] equal to the shortest path distance from s to v, for all vertices v, and sets parent variables to form a shortest path tree CSCE 411, Spring 2012: Set 8 64 BFS Running Time Initialization of each vertex takes O(V) time Every vertex is enqueued once and dequeued once, taking O(V) time When a vertex is dequeued, all its neighbors are checked to see if they are unvisited, taking time proportional to number of neighbors of the vertex, and summing to O(E) over all iterations Total time is O(V+E) CSCE 411, Spring 2012: Set 8 65 Depth-First Search Input: G = (V,E) for each vertex u in V do for each unvisited vertex u in V do mark u as unvisited call recursiveDFS(u) recursiveDFS(u): mark u as visited for each unvisited neighbor v of u do call recursiveDFS(v) CSCE 411, Spring 2012: Set 8 66 DFS Example start at a and consider in alphabetical order a b c d e f g h CSCE 411, Spring 2012: Set 8 67 Disconnected Graphs What if graph is disconnected or is directed? call DFS on several vertices to visit all vertices purpose of second for-loop in non-recursive wrapper a b d c e CSCE 411, Spring 2012: Set 8 68 DFS Tree Actually might be a DFS forest (collection of trees) Keep track of parents CSCE 411, Spring 2012: Set 8 69 Depth-First Search #2 Input: G = (V,E) for each vertex u in V do mark u as unvisited parent[u] := nil recursiveDFS(u): for each unvisited vertex u in V do parent[u] := u // a root call recursive DFS(u) CSCE 411, Spring 2012: Set 8 mark u as visited for each unvisited neighbor v of u do parent[v] := u call recursiveDFS(v) 70 More Properties of DFS Need to keep track of more information for each vertex: discovery time: when its recursive call starts finish time: when its recursive call ends CSCE 411, Spring 2012: Set 8 71 Depth-First Search #3 Input: G = (V,E) for each vertex u in V do mark u as unvisited parent[u] := nil recursiveDFS(u): time := 0 for each unvisited vertex u in V do parent[u] := u // a root call recursive DFS(u) mark u as visited time++ disc[u] := time for each unvisited neighbor v of u do CSCE 411, Spring 2012: Set 8 parent[v] := u call recursiveDFS(v) time++ fin[u] := time 72 Running Time of DFS initialization takes O(V) time second for loop in non-recursive wrapper considers each vertex, so O(V) iterations one recursive call is made for each vertex in recursive call for vertex u, all its neighbors are checked; total time in all recursive calls is O(E) Total time is O(V+E) CSCE 411, Spring 2012: Set 8 73 Minimum Spanning Tree 16 5 12 4 7 3 6 11 14 9 8 10 2 15 13 17 18 Given a connected undirected graph with edge weights, find subset of edges that spans all the nodes, creates no cycle, and minimizes sum of weights CSCE 411, Spring 2012: Set 4 74 Facts About MSTs There can be many spanning trees of a graph In fact, there can be many minimum spanning trees of a graph But if every edge has a unique weight, then there is a unique MST CSCE 411, Spring 2012: Set 4 75 Generic MST Algorithm input: weighted undirected graph G = (V,E,w) T := empty set while T is not yet a spanning tree of G find an edge e in E s.t. T U {e} is a subgraph of some MST of G add e to T return T (as MST of G) CSCE 411, Spring 2012: Set 9 76 Kruskal's MST algorithm 16 5 12 4 7 3 6 11 14 9 8 10 2 15 13 17 18 consider the edges in increasing order of weight, add in an edge iff it does not cause a cycle CSCE 411, Spring 2012: Set 4 77 Implementing Kruskal's Alg. Sort edges by weight efficient algorithms known How to test quickly if adding in the next edge would cause a cycle? use disjoint set data structure, later CSCE 411, Spring 2012: Set 4 78 Kruskal's Algorithm as a Special Case of Generic Algorithm Consider edges in increasing order of weight Add the next edge iff it doesn't cause a cycle At any point, T is a forest (set of trees); eventually T is a single tree CSCE 411, Spring 2012: Set 9 79 Another MST Algorithm Kruskal's algorithm maintains a forest that grows until it forms a spanning tree Alternative idea is keep just one tree and grow it until it spans all the nodes Prim's algorithm At each iteration, choose the minimum weight outgoing edge to add greedy! CSCE 411, Spring 2012: Set 4 80 Idea of Prim's Algorithm Instead of growing the MST as possibly multiple trees that eventually all merge, grow the MST from a single vertex, so that there is only one tree at any point. Also a special case of the generic algorithm: at each step, add the minimum weight edge that goes out from the tree constructed so far. CSCE 411, Spring 2012: Set 9 81 Prim's Algorithm input: weighted undirected graph G = (V,E,w) T := empty set S := {any vertex in V} while |T| < |V| - 1 do let (u,v) be a min wt. outgoing edge (u in S, v not in S) add (u,v) to T add v to S return (S,T) (as MST of G) CSCE 411, Spring 2012: Set 9 82 Prim's Algorithm Example 8 b 4 c d 9 2 11 a 7 i 4 7 8 h 6 1 g 2 CSCE 411, Spring 2012: Set 9 e 14 f 10 83 Implementing Prim's Algorithm How do we find minimum weight outgoing edge? First cut: scan all adjacency lists at each iteration. Results in O(VE) time. Try to do better. CSCE 411, Spring 2012: Set 9 84 Implementing Prim's Algorithm Idea: have each vertex not yet in the tree keep track of its best (cheapest) edge to the tree constructed so far. To find min wt. outgoing edge, find minimum among these values use a priority queue to store the best edge info (insert and extract-min operations) CSCE 411, Spring 2012: Set 9 85 Implementing Prim's Algorithm When a vertex v is added to T, some other vertices might have their best edges affected, but only neighbors of v add decrease-key operation to the priority queue v's best edge to Ti x Ti u Ti+1 v check if this edge is cheaper for w w x's best edge to Ti w's best edge to Ti CSCE 411, Spring 2012: Set 9 86 Details on Prim's Algorithm Associate with each vertex v two fields: best-wt[v] : if v is not yet in the tree, then it holds the min. wt. of all edges from v to a vertex in the tree. Initially infinity. best-node[v] : if v is not yet in the tree, then it holds the name of the vertex (node) u in the tree s.t. w(v,u) is v's best-wt. Initially nil. CSCE 411, Spring 2012: Set 9 87 Details on Prim's Algorithm input: G = (V,E,w) // initialization initialize priority queue Q to contain all vertices, using best-wt values as keys let v0 be any vertex in V decrease-key(Q,v0,0) // last line means change best-wt[v0] to 0 and adjust Q accordingly CSCE 411, Spring 2012: Set 9 88 Details on Prim's Algorithm while Q is not empty do u := extract-min(Q) // vertex w/ smallest best-wt if u is not v0 then add (u,best-node[u]) to T for each neighbor v of u do if v is in Q and w(u,v) < best-wt[v] then best-node[v] := u decrease-key(Q,v,w(u,v)) return (V,T) // as MST of G CSCE 411, Spring 2012: Set 9 89 Running Time of Prim's Algorithm Depends on priority queue implementation. Let Tins be time for insert Tdec be time for decrease-key Tex be time for extract-min Then we have |V| inserts and one decrease-key in the initialization: O(VTins+Tdec) |V| iterations of while one extract-min per iteration: O(VTex) total CSCE 411, Spring 2012: Set 9 90 Running Time of Prim's Algorithm Each iteration of while includes a for loop. Number of iterations of for loop varies, depending on how many neighbors the current vertex has Total number of iterations of for loop is O(E). Each iteration of for loop: one decrease key, so O(ETdec) total CSCE 411, Spring 2012: Set 9 91 Running Time of Prim's Algorithm O(V(Tins + Tex) + ETdec) If priority queue is implemented with a binary heap, then Tins = Tex = Tdec = O(log V) total time is O(E log V) (Think about how to implement decrease-key in O(log V) time.) CSCE 411, Spring 2012: Set 9 92 Shortest Paths in a Graph Let's review two important single-source shortest path algorithms: Dijkstra's algorithm Bellman-Ford algorithm CSCE 411, Spring 2012: Set 9 93 Single Source Shortest Path Problem Given: directed or undirected graph G = (V,E,w) and source vertex s in V Find: For each t in V, a path in G from s to t with minimum weight Warning! Negative weights are a problem: s 4 t 5 CSCE 411, Spring 2012: Set 9 94 Shortest Path Tree Result of a SSSP algorithm can be viewed as a tree rooted at the source Why not use breadth-first search? Works fine if all weights are the same: weight of each path is (a multiple of) the number of edges in the path Doesn't work when weights are different CSCE 411, Spring 2012: Set 9 95 Dijkstra's SSSP Algorithm Assumes all edge weights are nonnegative Similar to Prim's MST algorithm Start with source vertex s and iteratively construct a tree rooted at s Each vertex keeps track of tree vertex that provides cheapest path from s (not just cheapest path from any tree vertex) At each iteration, include the vertex whose cheapest path from s is the overall cheapest CSCE 411, Spring 2012: Set 9 96 Prim's vs. Dijkstra's 4 5 4 1 s 5 s 6 Prim's MST 1 6 Dijkstra's SSSP CSCE 411, Spring 2012: Set 9 97 Implementing Dijkstra's Alg. How can each vertex u keep track of its best path from s? Keep an estimate, d[u], of shortest path distance from s to u Use d as a key in a priority queue When u is added to the tree, check each of u's neighbors v to see if u provides v with a cheaper path from s: compare d[v] to d[u] + w(u,v) CSCE 411, Spring 2012: Set 9 98 Dijkstra's Algorithm input: G = (V,E,w) and source vertex s // initialization d[s] := 0 d[v] := infinity for all other vertices v initialize priority queue Q to contain all vertices using d values as keys CSCE 411, Spring 2012: Set 9 99 Dijkstra's Algorithm while Q is not empty do u := extract-min(Q) for each neighbor v of u do if d[u] + w(u,v) < d[v] then d[v] := d[u] + w(u,v) decrease-key(Q,v,d[v]) parent(v) := u CSCE 411, Spring 2012: Set 9 100 Dijkstra's Algorithm Example a 2 iteration b 12 8 10 6 c 3 4 d 2 4 e source is vertex a 0 1 2 3 4 5 Q abcde bcde cde de d Ø d[a] 0 0 0 0 0 0 d[b] ∞ 2 2 2 2 2 d[c] ∞ 12 10 10 10 10 d[d] ∞ ∞ ∞ 16 13 13 d[e] ∞ ∞ 11 11 11 11 9 CSCE 411, Spring 2012: Set 9 101 Running Time of Dijkstra's Alg. initialization: insert each vertex once O(V) iterations of while loop one extract-min per iteration => O(V Tex) for loop inside while loop has variable number of iterations… For loop has O(E) iterations total O(V Tins) one decrease-key per iteration => O(E Tdec) Total is O(V (Tins + Tex) + E Tdec) CSCE 411, Spring 2012: Set 9 102 Using Different Heap Implementations O(V(Tins + Tex) + ETdec) If priority queue is implemented with a binary heap, then Tins = Tex = Tdec = O(log V) total time is O(E log V) There are fancier implementations of the priority queue, such as Fibonacci heap: Tins = O(1), Tex = O(log V), Tdec = O(1) (amortized) total time is O(V log V + E) CSCE 411, Spring 2012: Set 9 103 Using Simpler Heap Implementations O(V(Tins + Tex) + ETdec) If graph is dense, so that |E| = (V2), then it doesn't help to make Tins and Tex to be at most O(V). Instead, focus on making Tdec be small, say constant. Implement priority queue with an unsorted array: Tins = O(1), Tex = O(V), Tdec = O(1) total is O(V2) CSCE 411, Spring 2012: Set 9 104 What About Negative Edge Weights? Dijkstra's SSSP algorithm requires all edge weights to be nonnegative even more restrictive than outlawing negative weight cycles Bellman-Ford SSSP algorithm can handle negative edge weights even "handles" negative weight cycles by reporting they exist CSCE 411, Spring 2012: Set 9 105 Bellman-Ford Idea Consider each edge (u,v) and see if u offers v a cheaper path from s compare d[v] to d[u] + w(u,v) Repeat this process |V| - 1 times to ensure that accurate information propgates from s, no matter what order the edges are considered in CSCE 411, Spring 2012: Set 9 106 Bellman-Ford SSSP Algorithm input: directed or undirected graph G = (V,E,w) //initialization initialize d[v] to infinity and parent[v] to nil for all v in V other than the source initialize d[s] to 0 and parent[s] to s // main body for i := 1 to |V| - 1 do for each (u,v) in E do // consider in arbitrary order if d[u] + w(u,v) < d[v] then d[v] := d[u] + w(u,v) parent[v] := u CSCE 411, Spring 2012: Set 9 107 Bellman-Ford SSSP Algorithm // check for negative weight cycles for each (u,v) in E do if d[u] + w(u,v) < d[v] then output "negative weight cycle exists" CSCE 411, Spring 2012: Set 9 108 Running Time of Bellman-Ford O(V) iterations of outer for loop O(E) iterations of inner for loop O(VE) time total CSCE 411, Spring 2012: Set 9 109 Bellman-Ford Example a 3 s 2 b —4 4 c 1 process edges in order (c,b) (a,b) (c,a) (s,a) (s,c) <board work> CSCE 411, Spring 2012: Set 9 110 Speeding Up Bellman-Ford The previous example would have converged faster if we had considered the edges in a different order in the for loop move outward from s If the graph is a DAG (no cycles), we can fully exploit this idea to speed up the running time CSCE 411, Spring 2012: Set 9 111