Download 7.7. Moment generating functions

1
7.7. Moment generating functions
Definition 1 We define the moment generating function MX by
MX (t) = EetX ,
provided this is finite. In the discrete case this is equal to
tx
x e p(x).
P
It two random variables have the same m.g.f. then they must have the same distribution of probability.
The moment generating function uniquely determines the distribution of a random variable.
Proposition. 1. If MX (t) = MY (t) < ∞ for all t in an interval, then X and Y have the same distribution.
We say these two random variables X and Y have identical distributions. Note that it would be wrong
to say they were equal.
Example 2.5-1 HT. The m.g.f. of X is Mx (t) = et ( 36 ) + e2t ( 62 ) + e3t ( 61 ). Find the p.m.f. of X.
et /2
1−et /2 ,
Example 2.5-2 HT. The m.g.f. of X is Mx (t) =
for t < ln 2. Find the p.m.f. of X.
(Note the the m.g.f. exists only when the sum of the series is finite, so sometime the range of t need to
be specified.)
We call Mx (t) the moment generating function because all of the moment of X can be obtained by
successively differentiating MX (t) and the evaluating the result at t = 0. For example.
d tX
Ee
dt
d
= E (etX )
dt
= EXetX
′
MX (t) =
mean and variance of X by m.g.f.
Setting t = 0, we have
′
MX (0) = EX
′′
MX (0) = EX 2
′
′′
′
r (0) = EX r . In particular, u = M (0) and σ 2 = EX 2 − [EX]2 = M (0) − [M (0)]2
In general MX
X
X
X
Let us compute the moment generating function for some of the distributions we have been working
with.
Version 2007 for Math 3 Probability I by PW Tsai
2
1. Bernoulli(p): pet + q, where q = 1 − p.
2. Binomial(n, p): using independence,
Eet
P
Xi
=E
Y
etXi =
Y
EetXi = (pet + q)n ,
where q = 1 − p and the Xi are independent Bernoulli’s. Or computer directly.
Find mean and variance by differentiating the m.g.f. at t = 0
3. Poisson(λ):
EetX =
X etk e−λ λk
X (λet )k
= e−λ
k!
k!
t
= e−λ eλe = eλ(e
t −1)
.
4. Geometric(p)
EetX =
if t < − ln q
pet
1 − qet
5. Negative binomial (r, p)
EetX =
if t < − ln q
h pet ir
1 − qet
6. Exponential(λ):
tX
Ee
=
Z
∞
etx λe−λx dx =
0
if t < λ and ∞ if t ≥ λ.
7. N (0, 1):
1
√
2π
Z
tx −x2 /2
e e
t2 /2
dx = e
1
√
2π
Z
λ
λ−t
2 /2
e−(x−t)
dx = exp{
8. N (µ, σ 2 ): Write X = µ + σZ. Then
2 /2
EetX = Eetµ etσZ = etµ e(tσ)
= exp µt +
9. Gamma(α, λ)
EetX =
if t < λ
λ α
λ−t
10. Chi-square(v)
EetX =
if t < 1/2
1 v/2
1 − 2t
Version 2007 for Math 3 Probability I by PW Tsai
σ 2 t2 .
2
t2
}.
2
3
6. Jointly Distributed Random Variables
We want to discuss collections of random variables (X1 , X2 , . . . , Xn ), which are known as random
vectors.
The joint distribution of a pair of random variables X and Y is the probability distribution over the plane
defined by
P(B) = P((X, Y ) ∈ B)
for subsets B of the plane.
In the discrete case, we can define the density p(x, y) = P(X = x, Y = y). Remember that here the
comma means “and.” (combined or joint outcome). The range of the joint outcome (X, Y ) is the set of
all ordered pairs (x, y) with x in the range of X and y in the range of Y , and satisfying
P(x, y) ≥ 0
X
P(x, y) = 1
all (x,y)
Example Roll two dice. Let X be the sum of the two dice, and Y be the minimum. Table the density
function of X and Y .
Example 1a Two different balls drawn at random without replacement. An urn with 3 red, 4 white
and 5 blue balls and we randomly select 3 balls from the urn. Let X and Y , respectively, be the number
of red and white balls chosen are randomly selected. Determine the joint probability mass function of X
and Y .
Example 2. If fX,Y (x, y) = ce−x e−2y for 0 < x < ∞ and x < y < ∞, what is c?
Answer. We use the fact that a density must integrate to 1. So
Z ∞Z ∞
ce−x e−2y dy dx = 1.
0
x
Recalling multivariable calculus, this is
Z
∞
0
c
1
ce−x e−2x dx = ,
2
6
so c = 6.
Joint distribution function of X and Y
For any two r.v. X and Y , the joint (cumulative) distribution function of X and Y is defined by
FX,Y (x, y) = P{X ≤ x, Y ≤ y},
Version 2007 for Math 3 Probability I by PW Tsai
−∞ < x, y < ∞
4
In the continuous case, this is
Z
x
−∞
Z
y
fX,Y (x, y)dy dx,
−∞
and so we have
∂2
F (x, y).
∂x∂y
f (x, y) =
The extension to n random variables is exactly similar.
Probabilities of events determined by X and Y
We have
P(a ≤ X ≤ b, c ≤ Y ≤ d) =
Z bZ
a
d
fX,Y (x, y)dy dx,
c
or
P((X, Y ) ∈ D) =
Z Z
fX,Y dy dx
D
when D is the set {(x, y) : a ≤ x ≤ b, c ≤ y ≤ d}. One can show this holds when D is any set. For
example, in discrete case
P(X < Y ) =
X
P(x, y) =
X X
P(x, y);
all x y:y>x
(x,y):x<y
in continuous case,
P(X < Y ) =
Z Z
fX,Y (x, y)dy dx.
{x<y}
Example 1c If fX,Y (x, y) = 2e−x e−2y for 0 < x < ∞ and 0 < y < ∞, what is P(X > 1, Y < 1),
P(X < Y ) and P(X < a)?
Example 1e The joint density of X and Y is given by

 e−(x+y) if 0 < x, y < ∞
f (x, y) =
 0
otherwise
Find the density function of the random variable X/Y .
marginal probability density function
If one has the joint density of X and Y , one can recover the densities of X and of Y :
Z ∞
Z ∞
fX,Y (x, y)dx.
fX,Y (x, y)dy, fY (y) =
fX (x) =
−∞
−∞
For discrete r.v. we have
P(X = x) =
X
P(X = x, Y = y),
P(Y = y) =
all y
Version 2007 for Math 3 Probability I by PW Tsai
X
all x
P(X = x, Y = y)
5
we call these the marginal probability mass function of X and Y respectively.
Example 4.1-4 HT The joint p.m.f of X and Y is given by

 x+y x = 1, 2, 3,
21
f (x, y) =
 0
otherwise
y = 1, 2
Find the marginal p.m.f. of X and Y .
Example 4.1-5 (HT) The joint p.m.f of X and Y is given by
f (x, y) =
xy 2
30
x = 1, 2, 3,
y = 1, 2
Find the marginal p.m.f. of X and Y .
Answer. fX (x) = (2x + 3)/21, x = 1, 2, 3. fY (y) = (6 + 3y)/21, y = 1, 2
Example 4.1-10 (HT) The joint p.d.f of X and Y is given by
f (x, y) = 2,
Find (a) P(0 ≤ X ≤
EX, EY, EY 2
1
2, 0
≤ Y ≤
1
2)
0≤x≤y≤1
(b) fX (x) and fY (y), the marginal p.d.f. of X and Y . (c)
Answer. (a) 1/4, (b) fX (x) = 2(1 − x), 0 ≤ x ≤ 1, fY (y) = 2y, 0 ≤ y ≤ 1. (c) 1/3, 2/3, 1/2.
Question: Suppose you know the distribution function of X and the distribution function of Y separately, does this mean that you know their joint distribution (X, Y ) or the distriubtion of X + Y ?
Example Sum of two draws. Let X be the result of the first draw from a box containing {1, 2, 3}. Let Y
the result of the second draw after the first ticket has been replaced. Let Z be the result of the second draw
without replacing the first ticket. (a) Calculate the distribution of X + Y . (b) What is the joint distribution of X and Y ? What is the marginal p.m.f.s of X and Y ? (c) What is the joint distribution of X and Z.
Example 1f multinomial distribution
A multivariate random vector is (X1 , . . . , Xr ) with
P(X1 = n1 , . . . , Xr = nr ) =
n!
pn1 · · · pnr r ,
n1 ! · · · nr ! 1
where n1 + · · · + nr = n and p1 + · · · pr = 1.
Version 2007 for Math 3 Probability I by PW Tsai
6
6.2. Independent random variables
In the discrete case we say X and Y are independent if P(X = x, Y = y) = P(X = x)P(Y = y) for
all x and y.
If X and Y are independent random variables, then every event determined by X is independent of every
event determined by Y :
P(X ∈ A, Y ∈ B) = P(X ∈ A)P(Y ∈ B)
for all pairs of subsets A, B of the reals. The left hand side is an abbreviation for
P({ω : X(ω) is in A and Y (ω) is in B})
and similarly for the right hand side.
In the discrete case, if we have independence,
pX,Y (x, y) = P(X = x, Y = y) = P(X = x)P(Y = y) = pX (x)pY (y).
In other words, the joint density pX,Y factors.
In the continuous case,
Z bZ d
fX,Y (x, y)dy dx = P(a ≤ X ≤ b, c ≤ Y ≤ d) = P(a ≤ X ≤ b)P(c ≤ Y ≤ d)
a
c
Z d
Z b
fY (y)dy.
fX (x)dx
=
a
c
Proposition. 1. Independent
If X and Y are independent random variables, then
fX,Y (x, y) = fX (x)fY (y),
or again called the joint density factors. Going the other way, one can also see that if the joint density
factors, then one has independence.
Example 1 Uniform on a square
If X and Y are two independent uniform (0,1) random variables, find (a) P(X 2 + Y 2 ≤ 1) ; (b)
P(X 2 + Y 2 ≤ 1 | X + Y ≥ 1) ; (c) P(Y ≤ X 2 ) ; (d) P(|X − Y | ≤ 0.5) ; (e) P(Y ≥ X | Y ≥ 1/2)
Answer. (a) π/4, (b) π/2 − 1, (c) 1/3 (d) 0.75, (e) 0.75.
Example 2c Two people try to meet at a place between 5:00 pm to 5:30 pm. Suppose that each person
arrives at a time distributed uniformly at random in this time interval, independent of the other, and waits
for the other at most 5 minutes. What is the probability that they meet?
Version 2007 for Math 3 Probability I by PW Tsai
7
Answer. 11/36
Example 2a. Two Binomial.
Z ∼ Binomial(n + m, p) (n + m independent trials). Let X be the number of successes in the first n
trials, and Y be the number of successes in the final m trials. Show X and Y are independent.
Example 2b. Two Poisson.
Suppose that the number of people that enter a post office on a given day is a Poisson random variable
with parameter λ. Show that if each person enters the post office is a male with probability p and a female
with probability1 − p, then the number of males and females entering the post office are independent
Poisson random variables with parameters λp and λ(1 − p), respectively.
Answer. Let X and Y be the number of males and females that enter the post office.
i+j
λ
(1) Use conditional probability to find P(X = i, Y = j) knowing that P(X + Y = i + j) = e−λ (i+j)!
(2) Compute marginal p.m.f. of P(X = i) and P(Y = j) from the joint p.m.f. in (1)
(3) Show independent
If the joint density factors, then one has independence.
Example 1a Two different balls drawn at random without replacement. An urn with 3 red, 4 white
and 5 blue balls and we randomly select 3 balls from the urn. Let X and Y , respectively, be the number
of red and white balls chosen are randomly selected. Determine whether X and Y are independent?
Example Two Exponentials let X and Y be independent exponential random variables with parameter
λ1 and λ2 , respectively. Calculate P(X < Y ).
λ1
λ1 +λ2
Example 2f The joint density of X and Y is given by

 6e−(2x+3y)
f (x, y) =
 0
if 0 < x, y < ∞
otherwise
Are the random variable independent? What if the joint p.d.f. is f (x, y) = 24xy
0 < x, y < 1, 0 <
x + y < 1 and is equal to 0 otherwise?
Example 2d. Buffon’s needle Suppose one has a floor made out of wood planks and one drops a needle
onto it. What is the probability the needle will intersect one of cracks? Suppose the needle is of length
L and the wood planks are D across, where L ≤ D.
Version 2007 for Math 3 Probability I by PW Tsai
8
Answer. Let X be the distance from the midpoint of the needle to the nearest crack and let Θ be the
angle the needle makes with the projected line of length X. Then X and Θ will be independent. X is
uniform on [0, D/2] and Θ is uniform on [0, π/2]. A little geometry shows that the needle will cross a
crack if L/2 > X/ cos Θ. We have fX,Θ =
4
πD
and so we have to integrate this constant over the set
where X < (L cos Θ)/2 and 0 ≤ Θ ≤ π/2 and 0 ≤ X ≤ D/2. The integral is
L
P{X < cos Θ} =
2
Z
0
π/2 Z (L cos θ)/2
0
4
2L
dx dθ =
.
πD
πD
Example Uniform on a triangle
Suppose (X, Y ) is uniformly distributed over the region {(x, y) : 0 < x < y < 1}.
(a) Find the joint p.d.f. of (X, Y ).
(b) Find the marginal density fX (x) and fY (y).
(c) Are X and Y independent?
(d) Find EX and EY .
(e) Find E(XY )
Answer. (a) f (x, y) = 2, 0 < x < y < 1. (c) No. (e)1/3.
Density of the sum of two independent r.v.s using moment generating functions
Proposition. 2. If X and Y are independent, then MX+Y (t) = MX (t)MY (t).
MX+Y (t) = EetX etY = EetX EetY = MX (t)MY (t).
Example The m.g.f. of X is MX (t) = 12 (et + e2t ) and the m.g.f. of Y is MY (t) = et ( 63 ) + e2t ( 62 ) +
e3t ( 16 ). Find the joint p.m.f. of X and Y.
6.3. Sums of independent random variables
Density of the sum of two random variables, assumed independent.
If X and Y are independent, then
Z Z
P(X + Y ≤ a) =
=
Z
fX,Y (x, y)dx dy =
{x+y≤a}
∞ Z a−y
fX (x)fY (y)dx dy
Z−∞ −∞
= FX (a − y)fY (y)dy.
Version 2007 for Math 3 Probability I by PW Tsai
Z Z
fX (x)fY (y)dx dy
{x+y≤a}
(1)
9
Differentiating with respect to a, we have
fX+Y (a) =
Z
fX (a − y)fY (y)dy.
The convolution formula is the special case of the formula for the density of X + Y when joint density
factors.
Example 3a 1. Uniform
Suppose X and Y are ind. r.v.s, both uniformly distributed on (0,1), i.e.,{(x, y) : 0 < x < 1, 0 < y < 1}.
Find the p.d.f. of X + Y .
Answer. X, Y ∼ U (0, 1) → X + Y has a triangular distribution.
Example 3b 2. Gamma
If X is a gamma with parameters s and λ and Y is a gamma with parameters t and λ, then straightforward
integration shows that X + Y is a gamma with parameters s + t and λ.
2. In particular, the sum of n independent exponentials with parameter λ is a gamma with parameters n
and λ.
Show by the convolution formula for the m.g.f.
Example normal (Z 2 to χ2 (1))
√
√
√
√
3. If Z is a N (0, 1), then FZ 2 (y) = P(Z 2 ≤ y) = P(− y ≤ Z ≤ y) = FZ ( y) − FZ (− y).
Differentiating shows that fZ 2 (y) = ce−y/2 (y/2)(1/2)−1 , or Z 2 is a gamma with parameters
4. So if Zi are independent N (0, 1)’s, then
Pn
2
i=1 Zi
1
2
and 12 .
is a gamma with parameters n/2 and 21 , i.e., a χ2n .
5. If Xi is a N (µi , σi2 ) and the Xi are independent, then some lengthy calculations show that
P
P
is a N ( µi , σi2 ).
Pn
i=1 Xi
Example 7h (394) What is the distribution of X + Y when X and Y are independent Normal random
variables with parameter (µ1 , σ12 ) and (µ2 , σ22 ), respectively.
If X and Y are independent normals, then −Y is also a normal with E(−Y ) = −EY and Var(−Y ) =
(−1)2 VarY = VarY , and so X − Y is also normal.
The analogue for discrete random variables is easier.
Example 3w 6. Poisson
If X and Y takes only nonnegative integer values, we have
P(X + Y = r) =
r
X
k=0
P(X = k, Y = r − k) =
r
X
k=0
Version 2007 for Math 3 Probability I by PW Tsai
P(X = k)P(Y = r − k).
10
In the case where X is a Poisson with parameter λ and Y is a Poisson with parameter µ, we see that
X + Y is a Poisson with parameter λ + µ.
Example 7g (p394) Poisson r.v.s What is the distribution of X + Y when X and Y are independent
Poisson random variables with parameter λ1 and λ2 , respectively
Example 7f binomial
What is the distribution of X + Y when X and Y are independent Binomial random variables with
parameter (n, p) and (m, p), respectively.
Note that it is not always the case that the sum of two independent random variables will be a random
variable of the same type.
Version 2007 for Math 3 Probability I by PW Tsai