Download randomized.analysis

1
Professor John Reif
Probability Theory:
(a) Random Variables: Binomial and Geometric
(b) Useful Probabilistic Bounds and Inequalities
Reading Selections:
BB Chapter 8
Handout--Appendix of Queuing
Systems, Vol. I by Kleinrock
2
a probability measure Prob
is a mapping from a set of events
to the reals such that
(1) for any event A
0  Prob(A)  1
(2) Prob (all possible events) = 1
(3) if A,B are mutually exclusive
events, then
Prob (A  B) = Prob (A) + Prob (B)
3
Conditional Probability
define Prob(A|B) = Prob (A  B)
Prob (B)
for Prob(B) > 0
Bayes' Theorem If A1 ,..., An are
mutually exclusive and contain all events
then Prob (Ai | B) =
Pi
n
Pj

j=1
where Pj = Prob(B|Aj)  Prob(Aj)
4
B
A1
...
A2
Random Variable A
(over real numbers)
Density Function fA(x)=Prob(A=x)
1
f (x)
A
FA (x)
0
x
prob Distribution Function
Ai
...
An
5
x
F A(x) = Prob (Ax) =


F (x)
1
A
0
x
If for Random Variables A,B
x
FA(x)  FB(x)
then
"A upper bounds B" and
"B lower bounds A"
f A(x) dx
6
F (X)
B
F (X)
A
X
FA(x) = Prob (A  x)
FB(x) = Prob (B  x)
Expectation of Random Variable A
7

E(A) = A =

-
x f A(x) dx
A is als o called "av erage of A"
and "mean of A" = A
A
note
FA( A) = 1
2
8
Variance of Random Variable A
2
A
2
= (A-A) = A
2
- (A)
w here 2nd moment
2
A
2

=

-
2
x f A(x) dx
9
example small
variance
f (x)
A
small
x
example
large
variance
f (x)
large
A
x


n'th Moments of Random Variable A
A
n

=

-
x
n
f A(x) dx
10
moment generating function

MA(s ) =

e
sx
-
f A(x) dx
( )
= E e
note
sA
s is a formal parameter
n
n
A =
d MA (s)
ds





n
s=0 
Discrete Random Variable A
11
density function
f (x) = prob (A=x)
A
1
2
3
4
5
6
. . .
7
x
distribution function
x
F (x) =
A
1
2
3
4

i=0
5
fA (i) = prob (A
  x)
6
7
8
9
...
12



Discrete Random Variable A
over nonnegative integers

expectation E(A) = A =

x=0

n
n' th moment A =

x=0
x
x f A(x)
n
f A(x)
probability generating function

G A(z) =

x=0
1st derivative
2nd derivative
z
x
f A(x) = E
(z )
'
GA (1) = A
"
2
GA (1) = A - A
A
13
2
"
'
v ariance rA = GA (1) + GA (1) 


( G (1) )

A,B independent if Prob(AB)=Prob(A)  Prob(B)
equivalent definition of independence
fAB (x) = fA (x)  fB(x)
MAB (s) = MA (s)  MB(s)
GAB (z) = GA (z)  GB(z)
If A1,...,An independent with same distribution
f A (x) = f A (x)
1
for i=1 , . . . , n
i
Then if B = A1 A2 An
2
'
A

14
f B (x) =
MB (s ) =
(
(
f A (x)
1
MA (s )
1
)
)
n
n
, G B (z) =

Combinatorics
n! = n(n-1)  21
= number of permutations of n objects
Stirling's formula
n! = f(n) (1+o(1))
n
where f(n) = n e
-n
2n
(
G A (z)
1
)
n
15
note tighter bound
f(n) e
1
(12n+1)
< n! < f(n) e
n!
= number of permutations of n
(n-k)!
object s taken k at a time
( )
n
k
n!
=
k! (n-k)!
= number of (unordered)
combinations of n objects
taken k at a time
1
12n
16
Bounds (due to Erdos & Spencer, p. 18)
2
-
( )
n
k
k

n e
3
k
k
2n 6n2
k!
for k = o

( )
n
3
4
Bernoulli Variable Ai is 1 with prob P
and 0 with prob 1-P
Binomial Variable B is sum of n independent
Bernoulli variables Ai each with some
probability p
procedure BINOMIAL with parameters n,p
begin
B0
for i=1 to n do
with probability P do B  B+1
output B
end
17
B 0
i  0

i i + 1
i>n
1-
yes
output B
P
P
BB+1
B is Binomial Variable with parameters n,p
18

mean
= n  p
v ariance
density fn =
2
 np (1- p )
Prob(B=x) =
()
n
 ( k) p
n
x
p
x
distribution fn = Prob(Bx) =
x
(1-p)
k
n-x
(1-p)
n-k
k=0
generating function
n
G(z) = (1-p+pz) =
n
z
k= 0
k
(
n
k
)
k
p (1-p)
n-k
19
interesting fact Prob(B=) = 
 

n
Poisson trial Ai is 1 with prob Pi
and 0 with prob 1-Pi
Suppose B' is the sum of
n independent Poisson trials Ai with
probability Pi for i>1,...,n
Hoeffding's Theorem B' is upper
by a Binomial Variable B with
n
parameters n, p w here p =
Pi

i=1
n
bound
20
F
B'
F
B
Geometric Variable V parameter p
x 0
Prob(V=x)
=
procedure GEOMETRIC
p(1-p)
x
parameter p
begin
V 0
loop: with probability p goto exit
21
V
V+1
V
goto loop
exit:
output
0
V
p
1-p
1-p
mean =
p
output V
V
V+1
generating function

G(Z) =

k=0
Z
k
p
(p(1-p) ) =
1-(1-p)Z
k
Probabilistic Inequalities
for Random Variable A
mean = A
2
2
v ariance  A (A)
2
22

fA
lower tail
upper tail
 
x
Markov Inequality (uses only mean)
Prob (A x)  
x
Chebychev Inequality (uses mean and variance)
2
Prob (|A-| ) 

2

example If B is Binomial with parameters n,p
np
Then Prob (Bx)  x
Prob (|B-np| ) 
Gaussian Density function
np (1-p)

2
23
2
1
(x) =
2
e
x
2
"Bell Shaped" Curve




Normal Distribution
x
 (X ) =

(Y) dY
-
Bounds (Feller, p. 175) x > 0,
(x)
( )
1
1
- 3
x
x
1 - (x) 
(x)
x
24
x [0, 1]
x
1
= x (1) (x) - x (0) =
2
2e
Let Sn be the sum of n independently
distributed variables A1,...,An
each with mean  and
n


variance
n
So S n has mean and v ariance 
2
Strong Law of Large Numbers
The probability dens ity function of
Tn =
(S n - )
limits as n

to normal dis tribution (x)
x
2
25
Hence Prob (|S n-| x) (x) as n 
s o Prob (|S n - | x) 2(1- (x))
2 (x)/x
(s ince 1- (x)  (x)/x)
Chernoff Bound (uses all moments)
of Random Variable A
Prob (Ax) 
=
e
(s )-s x
e
-s x
MA (s )
for s 0
where  (s ) = ln (MA(s ))
 e
(s ) - s ' (s )
26
(
by s etting
x = ' (s )
1s t derivative minimizes bounds
)
need moment generating function
Chernoff Bound of
Discrete Random Variable A
Prob (A x)  z
-x
G A( z ) for z
choose z=zo to minimize above bound
27
need Probability Generating function
G A(z) =

x 
( )
x
z f A(x) = E z
A
Chernoff Bounds for
Binomial B with parameters n,p
Abov e mean
x
28
n-x
Prob (B x) 
 e
x-
( )

x
 e
( ) ( )
x
-x -

x
n-
n-x
s ince
(
1
1 x
for x  e
)
x
<
e
2
Below Mean x 
n-x
Prob (B x) 
( ) ()
n-
n-x
x

x
Anguin-Valiant’s Bounds
for Binomial B with parameters n,p
x
29
Just above mean = np for 0<  <1
2
Prob (B  (1+))  e
Just below mean  for
- 2
0<  <1
2
- 
Prob (B   ( 1 -  ) )  e

e
-2
e


-

tails are bounded by Normal distributions
30
Binomial Variable X with
and expectation µ=pN
parameters p, N
By Chernoff, Bound for p≤1/2
Prob(X≥ N/2)<2N pN/2
°
Raghavan - Spencer bound
For any ∂>0,



e

Prob X  1+    
1 

 1   



in FOCS‘86.

