Download The Multivariate Normal Distribution • The univariate normal

The Multivariate Normal Distribution
• The univariate normal distribution can be characterized by:
X has the univariate normal distribution if its density
function fX (x) satisfies
log fX (x) = quadratic function of x.
• The multivariate normal distribution can be characterized
similarly.
1
Preliminaries
• The joint cumulative distribution function (cdf) of X1, X2, . . . , Xp
is
F1:p (x1, x2, . . . , xp) = Pr {X1 ≤ x1, X2 ≤ x2, . . . , Xp ≤ xp}
• If F (x1, x2, . . . , xp) is differentiable, the joint probability density function (pdf) is
∂ p F (x1 , x 2 , . . . , x p )
f1:p (x1, x2, . . . , xp) =
.
∂x1∂x2 . . . ∂xp
2
• For r < p, the marginal cdf of X1, X2, . . . , Xr is
F1:r (x1, x2, . . . , xr ) = Pr {X1 ≤ x1, X2 ≤ x2, . . . , Xr ≤ xr }
= Pr {X1 ≤ x1, X2 ≤ x2, . . . , Xr ≤ xr ,
o
Xr+1 ≤ ∞, . . . , Xp ≤ ∞
= F1:p (x1, x2, . . . , xr , ∞, . . . , ∞)
• The marginal pdf is
f1:r (x1, x2, . . . , xr )
=
Z ∞
−∞
...
Z ∞
−∞
f1:p x1, x2, . . . , xr , ur+1, . . . , up dur+1 . . . dup
3
Independence
• The variables X1, X2, . . . , Xr are indendepent of Xr+1, Xr+2, . . . , Xp
if
F1:p (x1, x2, . . . , xp) =
F1:r (x1, x2, . . . , xr ) × F(r+1):p xr+1, xr+2, . . . , xp
• If the pdf f1:p (x1, x2, . . . , xp) exists, then also
f1:p (x1, x2, . . . , xp) =
f1:r (x1, x2, . . . , xr ) × f(r+1):p xr+1, xr+2, . . . , xp
4
Conditional Pdf
• The conditional pdf of X1, X2, . . . , Xr , given
Xr+1 = xr+1, Xr+2 = xr+2, . . . , Xp = xp is
f1:r|(r+1):p x1, x2, . . . , xr |xr+1, xr+2, . . . , xp
=
f1:p x1, x2, . . . , xr , xr+1, xr+2, . . . , xp
f(r+1):p xr+1, xr+2, . . . , xp
• Note: the conditioning event has probability zero. In general,
this conditional pdf must be defined as a Radon-Nikodym
derivative. The expression may be derived as a limit, on the
assumption that the pdfs are all continuous and the denominator is non-zero.
5
Characterizing the Multivariate Normal Distribution
• (-2 times) the logarithm of the pdf is a quadratic:
− 2 log f1:p (x1, x2, . . . , xp) =
p
X
ai,ix2
i +2
i=1
p−1
X
p
X
ai,j xixj + 2
i=1 j=i+1
p
X
c i xi + d
i=1
• In matrix notation:
−2 log f (x) = x0Ax + 2c0x + d
6
• For f (x) to be integrable, it must converge to 0 as x → ∞,
so A must be positive definite.
• In particular, A−1 exists.
• Write b = A−1c. Then
−2 log f (x) = (x − b)0A(x − b) + d∗,
where d∗ = d − b0Ab.
• Equivalently
1
f (x) = K exp − (x − b)0A(x − b)
2
7
Standardization
• Choleski: A = LL0 for lower triangular L.
• Define a new multivariate random variable Y = L0(X − b).
• The pdf of Y is then
0−1
fY (y) = fX b + L y × J(y),
where J(y) is the Jacobian
∂ x J(y) = det
∂y
= | det L0−1|
q
=
det A−1
8
• That is,
1
fY (y) = K det A−1 exp − y0y
q
2
1
= K det A−1 exp − (y12 + y22 + . . . yp2)
2
q
q
p
Y
= K det 2π A−1
φ(yi),
i=1
where
2 /2
−1/2
−y
φ(y) = (2π)
e
is the standard (univariate) normal density.
9
√
• Since both√fY (y) and φ(y) integrate to 1, K det 2π A−1 = 1,
or K = 1/ det 2π A−1.
• So the pdf of X is
1
exp − (x − b)0A(x − b)
f (x) = √
2
det 2π A−1
1
• More importantly, Y1, Y2, . . . , Yp are mutually independent,
and each has the standard normal distribution.
10
Moments
• Definition: the expected value of a random vector or matrix
is the vector or matrix of elementwise expected values.
• Because Y1, Y2, . . . , Yp are mutually independent N (0, 1), their
mean vector is
E(Y) = 0
and their covariance matrix is
C(Y) , E [Y − E(Y)] [Y − E(Y)]
n
0
o
=I
11
• Since X = b + L0−1Y,
E(X) = b + L0−1E(Y) = b,
• Similarly
C(X) = L0−1C(Y)L−1 = (LL0)−1 = A−1
• If we write µ for
the pdf of X is
E(X)
and Σ for
C(X),
then A = Σ−1, and
1
exp − (x − µ)0Σ−1(x − µ) , n(x|µ, Σ)
fX (x) = √
2
det 2π Σ
1
12