Download CSE 638: Advanced Algorithms Supplemental Material ( High

CSE 638: Advanced Algorithms
Supplemental Material
( High Probability Bounds )
Rezaul A. Chowdhury
Department of Computer Science
SUNY Stony Brook
Spring 2013
Markov’s Inequality
Theorem 1: Let be a random variable that assumes only
nonnegative values. Then for all 0,
Pr .
Proof: For 0, let
Since 0, .
1if ;
0otherwise.
We also have, Pr 1 Pr .
Then Pr .
Example: Coin Flipping
Let us bound the probability of obtaining more than
sequence of fair coin flips.
Let
1ifthe!thcoinflipisheads;
0otherwise.
Then the number of heads in flips, ∑)* .
We know, Pr 1 *
.
+
Hence, ∑)* +.
Then applying Markov’s inequality,
Pr ⁄
⁄+
⁄
+
.
heads in a
Chebyshev’s Inequality
Theorem 2: For any 0,
./0 Pr - .
+
+
Proof: Observe that Pr - Pr - + .
Since - + is a nonnegative random variable, we can use
Markov’s inequality,
Pr - +
+
1 2
2
345 2
.
Example: 6 Fair Coin Flips
1ifthe!thcoinflipisheads;
0otherwise.
Then the number of heads in flips, ∑)* .
We know, Pr 1 Then ./0 Hence, ∑)* +
*
+
- +
+
and +
*
+
*
-
and ./0 *
.
∑)* ./0
*
.
+
Then applying Chebyshev’s inequality,
Pr Pr - 345 ⁄ 2
⁄
⁄ 2
.
.
Binomial Distribution
The binomial distribution is the
discrete probability distribution
of the #successes in a sequence
of independent yes/no
experiments (i.e., Bernoulli
trials), each of which succeeds
with probability 8.
Probability mass function:
9 :; , 8 Pr : :
8< 1 - 8
Source
Prof. Nick Harvey
( UBC )
RED: Binomial distribution with 20 and 8 1<
,
0:
Cumulative distribution function:
<
= :; , 8 Pr : >
)?
8 1-8
!
1
,
0:
*
+
Approximating with Normal Distribution
Normal distribution with mean @ and variance A + is given by:
9 C; @, A + 1
A 2D
E
1
* F1G 2
+ H
,
C∈ℜ
For fixed 8 as increases the binomial distribution with parameters
and 8 is well approximated by a normal distribution with @ 8
and A + 8 1 - 8 .
RED: Binomial distribution with
20 and 8 *
+
BLUE: Normal distribution with
@ 8 10
and A + 8 1 - 8 5
Source
Prof. Nick Harvey
( UBC )
Approximating with Normal Distribution
Normal distribution with mean @ and variance A + is given by:
9 C; @, A + 1
A 2D
E
1
* F1G 2
+ H
,
C∈ℜ
The probability that a normally distributed random variable lies in
the interval J∞, CL is given by:
= C; @, A
+
where, erf N *
+
1 M erf
F1G
H +
+ T 1Q 2
P E RS.
O ?
,
But erf N cannot be expressed
in closed form in terms of
elementary functions, and
hence difficult to evaluate.
Source
Prof. Nick Harvey
( UBC )
Approximating with Poisson Distribution
Poisson distribution with mean @ 0 is given by:
@< E 1G
9 :; @ ,
:!
: 0,1,2, …
If 8 is fixed and increases the binomial distribution with
parameters and 8 is well approximated by a Poisson distribution
with @ 8.
RED: Binomial distribution with
50 and 8 BLUE: Poisson distribution with
@ 8 200
Observe that the asymmetry in the plot
cannot be well approximated by a
symmetric normal distribution.
Source
Prof. Nick Harvey
( UBC )
Preparing for Chernoff Bounds
Lemma 1: Let * , … , be independent Poisson trials, that is, each
is a 0-1 random variable with Pr 1 8 for some 8 . Let
∑)* and @ . Then for any S 0,
E Q
E
W X 1* G .
Proof: E QY 8 E QZ* M 1 - 8 E QZ? 8 E Q M 1 - 8
1 M 8 EQ - 1
\
But for any [, 1 M [ E . Hence, E
Now, E
Q
E
b
Q ∑^
Y_` Y
)*
E
QY
E
]Y W X 1*
∏)* E QY ∏)* E QY
]Y W X 1*
E
W X 1* ∑^
Y_` ]Y
But, @ ∑)* ∑)* ∑)* 8 .
Hence, E Q
E
W X 1* G .
.
Chernoff Bound 1
Theorem 3: Let * , … , be independent Poisson trials, that is,
each is a 0-1 random variable with Pr 1 8 for some 8 .
Let ∑)* and @ . Then for any 0,
Wc
*d `ec
Pr 1 M @ G
.
Proof: Applying Markov’s inequality for any S 0,
Pr 1 M @ Pr E Q E Q
fX g` h
W
W X `ec h
*d G
E Q
Q *d G
E
[ Lemma 1 ]
Setting S ln 1 M 0, i.e., E Q 1 M , we get,
Pr 1 M @ Wc
*d `ec
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
Hence, 9 l first decreases and then increases over 0,1 .
Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 .
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
Hence, 9 l first decreases and then increases over 0,1 .
Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 .
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
Hence, 9 l first decreases and then increases over 0,1 .
Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 .
Since 9 0 0, it follows that 9 0 in that interval.
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
Hence, 9 l first decreases and then increases over 0,1 .
Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 .
Since 9 0 0, it follows that 9 0 in that interval.
G
.
Chernoff Bound 2
Theorem 4: For 0 i i 1, Pr 1 M @ E
hc2
1 j
.
Proof: From Theorem 3, for 0, Pr 1 M @ We will show that for 0 i i 1,
Wc
*d `ec
E
c2
1j
⇒ - 1 M ln 1 M That is, 9 - 1 M ln 1 M We have, 9′ - ln 1 M Observe that 9′′ i 0 for 0
2
M 2
- 0
+
M ,and 9′′ *
+ , and 9′′
Wc
*d `ec
*
+
- *d M *
+
0 for .
Hence, 9 l first decreases and then increases over 0,1 .
Since 9 l 0 0 and 9 l 1 i 0, we have 9 l 0 over 0,1 .
Since 9 0 0, it follows that 9 0 in that interval.
G
.
Chernoff Bound 3
Corollary 1: For 0 i m i @,Pr @ M m E
n2
1jh
.
Proof: From Theorem 2, for 0 i i 1, Pr 1 M @ i E
Setting m @, we get, Pr @ M m E
n2
1jh
hc2
1
j
for 0 i m i @.
.
Example: 6 Fair Coin Flips
1ifthe!thcoinflipisheads;
0otherwise.
Then the number of heads in flips, ∑)* .
We know, Pr 1 *
.
+
Hence, @ ∑)* +.
*
+
Now putting in Chernoff bound 2, we have,
Pr E
^
2o
1
*
^
W 2o
.
Chernoff Bounds 4, 5 and 6
W gc
*1 `gc
Theorem 5: For 0 i i 1, Pr 1 - @ Theorem 6: For 0 i i 1, Pr 1 - @ E
Corollary 2: For 0 i m i @,Pr @ - m E
hc2
1 2
n2
12h
.
.
G
.
Chernoff Bounds
Lower Tail
Upper Tail
E 1
1 - *1
p i q i r: Pr 1 - @ p i q i r: Pr 1 - @ E
p i t i u: Pr @ - m v2
1
E +G
1
G 2
+
G
q p: Pr 1 M @ E
1M
p i q i r: Pr 1 M @ E
p i t i u:Pr @ M m E
Source
greedyalgs.info
1
1
hc2
j
n2
jh
G
*d