Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Probability in Computing LECTURE 4: CONDITIONAL PROBABILITY, APPLICATIONS, GEOMETRIC DISTRIBUTION © 2010, Quoc Le & Van Nguyen Probability for Computing 1 Agenda Application: Verification of Matrix Multiplication Application: Randomized Min-Cut Geometric Distribution Coupon Collector’s Problem © 2010, Quoc Le & Van Nguyen Probability for Computing 2 Application: Verifying Matrix Multiplication Consider matrix multiplication AB = C (integers modulo 2) Simple algorithm takes O(n3) operations. Want to check if a given matrix multiplication program works correctly Randomized Algorithm: Choose a random vector r = (r1, r2, …, rn) in {0,1}n. Compute A(Br) and Cr then comparer the two values: if equal return yes AB=C, else no. Note on the above randomized algorithm: 1-side error Complexity = O(n2) Accuracy depends on P(ABr = Cr) when AB!=C © 2010, Quoc Le & Van Nguyen Probability for Computing 3 Analysis of P(ABr = Cr) Choosing r randomly is equivalent to choosing ri randomly and independently. (1) Let D = AB – C ≠ 0. Since Dr = 0, there must be some non-zero entry. Let that be d11. Dr = 0 ∑d1jrj = 0 r1 = -∑d1jrj / d11. Since r1 can take 2 values, combine with (1), we have ABr = Cr with probability of at most ½ Refer to book for formal proof (using Law of Total Probability) Principle of Deferred Decisions: when there are several random variables, it often helps to think of some of them as being set at one point in the algorithm with the rest of them being left random (or deferred) until some further point in the analysis. We can attempt this verification k times to obtain accurate answer with p = 2-k and efficiency = O(kn2) = O(n2) © 2010, Quoc Le & Van Nguyen Probability for Computing 4 Theorems Law of Total Probability: Assume E1, E2, …, En be mutually disjoint events in the sample space Ω and union of Ei = Ω. Then Pr(B) = ∑Pr(B and Ei) = ∑Pr(B|Ei)Pr(Ei) Bayes’ Law: Assume E1, E2, …, En be mutually disjoint events in the sample space Ω and union of Ei = Ω. Then Pr(Ej|B) = Pr(Ej and B)/Pr(B) = Pr(B|Ej)Pr(Ej) / ∑Pr(B|Ei)Pr(Ei) Notice the model transformation from prior probability to posterior probability. © 2010, Quoc Le & Van Nguyen Probability for Computing 5 Gradual Change in Our Confidence in Algorithm Correctness In matrix verification case: E = the identify is correct B = test returns that the identity is correct Prior assumption: Identity = ½ How does this assumption change after each run? We start with Pr(E) = Pr(Ec) = ½ Since the test has error bounded by ½, Pr(B|Ec) ≤ ½. Also, Pr(B|E) = 1 Now by Bayes’ Law: Pr(E|B) = Pr(B|E)Pr(E) / {Pr(B|E)Pr(E)+ Pr(B|Ec)Pr(Ec)}≥ ½ / {1.½ + ½. ½} = 2/3 © 2010, Quoc Le & Van Nguyen Probability for Computing 6 Gradual Change in Our Confidence in Algorithm Correctness The prior model is revised: Pr(E) ≥ 2/3 and Pr(Ec) ≤ 1/3. Applying Bayes’ Law again will yeild Pr(E|B) ≥ 4/5 In general, at ith iteration, Pr(E|B) ≥1 – 1/(2i+1) After 100 calls, test returns that identity is correct, then our confidence in the correctness of this identity is at least 1 – 1/ 2100+1) © 2010, Quoc Le & Van Nguyen Probability for Computing 7 Application: Randomized Min Cut Cut-set: Set of edges whose removal breaks the graph into two or more connected components. Min-cut: Cut-set with minimum cardinality. Applications: Network reliability. Clustering problems Al-Qaeda © 2010, Quoc Le & Van Nguyen Probability for Computing 8 Example F A B C © 2010, Quoc Le & Van Nguyen E G D Probability for Computing 9 Example F A B C E G D E1 = BE © 2010, Quoc Le & Van Nguyen Probability for Computing 10 Karger algorithm Edge contraction (collapsing an edge): To collapse edge {u,v}, we Create a new node uv Replace any edge of form u, w or v, w with new edge uv, w Delete original vertices u and v. Resulting graph is denoted G/{u,v}. Repeat until 2 nodes left (n-2 iterations): Choose edge at random “Contract” edge. Take all the edges between them as min-cut © 2010, Quoc Le & Van Nguyen Probability for Computing 11 Example F A B C © 2010, Quoc Le & Van Nguyen E G D Probability for Computing 12 Example F A B C © 2010, Quoc Le & Van Nguyen contract E G D Probability for Computing 13 Example F A B C contract E G D FG A B E C D © 2010, Quoc Le & Van Nguyen Probability for Computing 14 Example F A B C contract E G D FG A B E C D © 2010, Quoc Le & Van Nguyen Probability for Computing 15 contract Example F A B C contract E G D FGD FG A A B B E C D C © 2010, Quoc Le & Van Nguyen E Probability for Computing contract 16 Analysis Let k be the size of the min-cut set of G We want to compute the probability of finding one such set C. C partition V (set of vertices) in to S and V-S If the algorithm never choose an edge in C in its n-2 iterations, then the algorithm will return C as minimum cut-set. Let Ei = edge contracted in iteration i is not in C Let Fi = Union of Ej (j = 1i) = no edge of C was contracted in the first i iterations. We need to compute Pr(Fn-2) © 2010, Quoc Le & Van Nguyen Probability for Computing 17 Analysis All vertices have degree k or larger graph must have ≥ nk/2 edges. Pr(F1) = Pr(E1) ≥ 1 – k/ (nk/2) = 1 – 2/n Conditionally: Pr(E2|F1) ≥ 1 – k/ (k(n-1)/2) = 1 – 2/(n-1). Similarly: Pr(Ei|Fi-1) ≥ 1 – 2/(n-i+1) Therefore: Pr(Fn-2) = Pr(En-2|Fn-3)* Pr(Fn-3)+ Pr(En-2|Fn-3c)* Pr(Fn-3c) = Pr(En-2|Fn-3)* Pr(Fn-3) since Pr(En-2|Fn-3c)=0 So, Pr(Fn-2) = Pr(En-2|Fn-3)* Pr(Fn-3) = Pr(En-2|Fn-3)* Pr(En-3|Fn-4) * Pr(Fn4) = Pr(En-2|Fn-3)* Pr(En-3|Fn-4) *… * Pr(E2|F1) * Pr(F1) = 2/n(n-1). © 2010, Quoc Le & Van Nguyen Probability for Computing 18 What’s next Karger: Use of idea of amplification – Run algorithm many times and return the smallest guess. Our probability of success ≥ 1 – ( 1 – 2/n(n-1) )N ≥ 1 – e2N/n(n-1). (due to 1-x ≤e-x) Choose N = c(n choose 2) ln(n), for some constant c, then it is correct with probability at least 1 – 1/nc. Complexity = O(n4logn). We can reduce the time complexity by an order of n2 to obtain O(n2(logn)3) http://www.cs.dartmouth.edu/~ac/Teach/CS105Winter05/Handouts/05-mincut.pdf © 2010, Quoc Le & Van Nguyen Probability for Computing 19 Geometric Distribution Flip a coin until it lands on head. What is the distribution of the number of flips? Perform a sequence of independent trials until the first success, where each trial succeeds with prob. = p. Def: A geometric random variable X with parameter p is given by the following probability distribution: Pr(X=n)=(1-p)n-1.p © 2010, Quoc Le & Van Nguyen Probability for Computing 20 Properties Lemma 1: (Memory-less): Pr(X=n+k|X>k) = Pr(X=n) Lemma 2: Let X be a discrete random variable that takes on only non-negative integer values. Then: E[X] = ∑Pr(X≥i) (i = 1 ∞) For a geometric random variable X(p), Pr(X ≥i) = (1-p)i-1 E[X] = 1/p. © 2010, Quoc Le & Van Nguyen Probability for Computing 21 Coupon Collector Problem Problem: Suppose that each box of cereal contains one of n different coupons. Once you obtain one of every type of coupon, you can send in for a prize. Question: How many boxes of cereal must you buy before obtaining at least one of every type of coupon. Let X be the number of boxes bought until at least one of every type of coupon is obtained. Let Xi be number of boxes bought while you had exactly i-1 different coupons. Then: X = ∑Xi. © 2010, Quoc Le & Van Nguyen Probability for Computing 22 Coupon Collector’s Problem Probability of obtaining a new coupon when i -1 coupons have been found: pi = 1 – (i-1)/n. Hence, E[Xi] = 1/pi = n/(n-i+1) E[X] = ∑E[Xi] = n ∑1/i = nH(n) (harmonic number) = nln(n). © 2010, Quoc Le & Van Nguyen Probability for Computing 23 Application: Packet Sampling Sampling packets on a router with probability p The number of packets transmitted after the last sampled packet until and including the next sampled packet is geometrically distributed. From the point of destination host, determining all the routers on the path is like a coupon collector’s problem. If there’s n routers, then the expected number of packets arrived before destination host knows all of the routers on the path = nln(n). © 2010, Quoc Le & Van Nguyen Probability for Computing 24