Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
CSE 638: Advanced Algorithms Supplemental Material ( High Probability Bounds ) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2013 Markov’s Inequality Theorem 1: Let be a random variable that assumes only nonnegative values. Then for all 0, Pr . Proof: For 0, let Since 0, . 1if ; 0otherwise. We also have, Pr 1 Pr . Then Pr . Example: Coin Flipping Let us bound the probability of obtaining more than sequence of fair coin flips. Let 1ifthe!thcoinflipisheads; 0otherwise. Then the number of heads in flips, ∑)* . We know, Pr 1 * . + Hence, ∑)* +. Then applying Markov’s inequality, Pr ⁄ ⁄+ ⁄ + . heads in a Chebyshev’s Inequality Theorem 2: For any 0, ./0 Pr - . + + Proof: Observe that Pr - Pr - + . Since - + is a nonnegative random variable, we can use Markov’s inequality, Pr - + + 1 2 2 345 2 . Example: 6 Fair Coin Flips 1ifthe!thcoinflipisheads; 0otherwise. Then the number of heads in flips, ∑)* . We know, Pr 1 Then ./0 Hence, ∑)* + * + - + + and + * + * - and ./0 * . ∑)* ./0 * . + Then applying Chebyshev’s inequality, Pr Pr - 345 ⁄ 2 ⁄ ⁄ 2 . . Binomial Distribution The binomial distribution is the discrete probability distribution of the #successes in a sequence of independent yes/no experiments (i.e., Bernoulli trials), each of which succeeds with probability 8. Probability mass function: 9 :; , 8 Pr : : 8< 1 - 8 Source Prof. Nick Harvey ( UBC ) RED: Binomial distribution with 20 and 8 1< , 0: Cumulative distribution function: < = :; , 8 Pr : > )? 8 1-8 ! 1 , 0: * + Approximating with Normal Distribution Normal distribution with mean @ and variance A + is given by: 9 C; @, A + 1 A 2D E 1 * F1G 2 + H , C∈ℜ For fixed 8 as increases the binomial distribution with parameters and 8 is well approximated by a normal distribution with @ 8 and A + 8 1 - 8 . RED: Binomial distribution with 20 and 8 * + BLUE: Normal distribution with @ 8 10 and A + 8 1 - 8 5 Source Prof. Nick Harvey ( UBC ) Approximating with Normal Distribution Normal distribution with mean @ and variance A + is given by: 9 C; @, A + 1 A 2D E 1 * F1G 2 + H , C∈ℜ The probability that a normally distributed random variable lies in the interval J∞, CL is given by: = C; @, A + where, erf N * + 1 M erf F1G H + + T 1Q 2 P E RS. O ? , But erf N cannot be expressed in closed form in terms of elementary functions, and hence difficult to evaluate. Source Prof. Nick Harvey ( UBC ) Approximating with Poisson Distribution Poisson distribution with mean @ 0 is given by: @< E 1G 9 :; @ , :! : 0,1,2, … If 8 is fixed and increases the binomial distribution with parameters and 8 is well approximated by a Poisson distribution with @ 8. RED: Binomial distribution with 50 and 8 BLUE: Poisson distribution with @ 8 200 Observe that the asymmetry in the plot cannot be well approximated by a symmetric normal distribution. Source Prof. Nick Harvey ( UBC ) Preparing for Chernoff Bounds Lemma 1: Let * , … , be independent Poisson trials, that is, each is a 0-1 random variable with Pr 1 8 for some 8 . Let ∑)* and @ . Then for any S 0, E Q E W X 1* G . Proof: E QY 8 E QZ* M 1 - 8 E QZ? 8 E Q M 1 - 8 1 M 8 EQ - 1 \ But for any [, 1 M [ E . Hence, E Now, E Q E b Q ∑^ Y_` Y )* E QY E ]Y W X 1* ∏)* E QY ∏)* E QY ]Y W X 1* E W X 1* ∑^ Y_` ]Y But, @ ∑)* ∑)* ∑)* 8 . Hence, E Q E W X 1* G . . Chernoff Bound 1 Theorem 3: Let * , … , be independent Poisson trials, that is, each is a 0-1 random variable with Pr 1 8 for some 8 . Let ∑)* and @ . Then for any 0, Wc *d `ec Pr 1 M @ G . Proof: Applying Markov’s inequality for any S 0, Pr 1 M @ Pr E Q E Q fX g` h W W X `ec h *d G E Q Q *d G E [ Lemma 1 ] Setting S ln 1 M 0, i.e., E Q 1 M , we get, Pr 1 M @ Wc *d `ec G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . Hence, 9 l first decreases and then increases over 0,1 . Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 . G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . Hence, 9 l first decreases and then increases over 0,1 . Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 . G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . Hence, 9 l first decreases and then increases over 0,1 . Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 . Since 9 0 0, it follows that 9 0 in that interval. G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . Hence, 9 l first decreases and then increases over 0,1 . Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 . Since 9 0 0, it follows that 9 0 in that interval. G . Chernoff Bound 2 Theorem 4: For 0 i i 1, Pr 1 M @ E hc2 1 j . Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1, Wc *d `ec E c2 1j ⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0 2 M 2 - 0 + M ,and 9′′ * + , and 9′′ Wc *d `ec * + - *d M * + 0 for . Hence, 9 l first decreases and then increases over 0,1 . Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 . Since 9 0 0, it follows that 9 0 in that interval. G . Chernoff Bound 3 Corollary 1: For 0 i m i @,Pr @ M m E n2 1jh . Proof: From Theorem 2, for 0 i i 1, Pr 1 M @ i E Setting m @, we get, Pr @ M m E n2 1jh hc2 1 j for 0 i m i @. . Example: 6 Fair Coin Flips 1ifthe!thcoinflipisheads; 0otherwise. Then the number of heads in flips, ∑)* . We know, Pr 1 * . + Hence, @ ∑)* +. * + Now putting in Chernoff bound 2, we have, Pr E ^ 2o 1 * ^ W 2o . Chernoff Bounds 4, 5 and 6 W gc *1 `gc Theorem 5: For 0 i i 1, Pr 1 - @ Theorem 6: For 0 i i 1, Pr 1 - @ E Corollary 2: For 0 i m i @,Pr @ - m E hc2 1 2 n2 12h . . G . Chernoff Bounds Lower Tail Upper Tail E 1 1 - *1 p i q i r: Pr 1 - @ p i q i r: Pr 1 - @ E p i t i u: Pr @ - m v2 1 E +G 1 G 2 + G q p: Pr 1 M @ E 1M p i q i r: Pr 1 M @ E p i t i u:Pr @ M m E Source greedyalgs.info 1 1 hc2 j n2 jh G *d