Download Joint Moments of Random Variables

Multiple Random Variables
In many applications we have to deal with many random variables. For example, in the
navigation problem, the position of a space craft is represented by three random variables
denoting the x, y and z coordinates. The noise affecting the R, G, B channels of colour
video may be represented by three random variables. In such situations, it is convenient to
define the vector-valued random variables where each component of the vector is a
random variable.
In this lecture, we extend the concepts of joint random variables to the case of multiple
random variables. A generalized analysis will be presented for n random variables
defined on the same sample space.
Joint CDF of n random variables
Consider n random variables X1 , X 2 ,.., X n defined on the same probability space
( S , F , P). We define the random vector X as,
 X1 
X 
 2
X  . 
 
 . 
 X n 
or
X '   X1
X2 . . Xn 
where ' indicates the transpose operation.
Thus an n  dimensional random vector X is defined by the mapping X : S  R n . A
particular value of the random vector X is denoted by
x=[ x1 x2 .. xn ]'
The CDF of the random vector X is defined as the joint CDF of X1 , X 2 ,.., X n . Thus
FX1 , X 2 ,.., X n ( x1 , x2 ,..xn )  FX ( x )
 P ({X1  x1 , X 2  x2 ,..X n  xn })
Some of the most important properties of the joint CDF are listed below. These properties
are mere extensions of the properties of two joint random variables.
Properties of the joint CDF of n random variables
(a) FX1 , X 2 ,.. X n ( x1 , x2 ,..xn ) is a non-decreasing function of each of its arguments.
(b) FX1 , X 2 ,.. X n (, x2 ,..xn )  FX1 , X 2 ,.. X n ( x1 , ,..xn )  ...  FX1 , X 2 ,.. X n ( x1 , x2 ,..  )  0
(c) FX1 , X 2 ,.. X n (, ,.., )  1
(d) FX1 , X 2 ,.. X n ( x1 , x2 ,..xn ) is right-continuous in each of its arguments.
1
(e) The marginal CDF of a random variable X i is obtained from FX1 , X 2 ,.. X n ( x1 , x2 ,..xn ) by
letting all random variables except X i tend to . Thus
FX1 ( x1 )  FX1 , X 2 ,.. X n ( x1 , ,.., ),
FX 2 ( x2 )  FX1 , X 2 ,.. X n (, x2 ,.., )
and so on.
Joint pmf of n discrete random variables
Suppose X is a discrete random vector defined on the same probability space
( S , F , P ). Then X is completely specified by the joint probability mass function
PX1 , X 2 ,.., X n ( x1 , x2 ,..xn )  PX (x)
 P({ X1  x1 , X 2  x2 ,.. X n  xn })
Given PX1 , X 2 ,.., X n ( x1 , x2 ,..xn ) we can find the marginal probability mass function
pX1  x1   .... pX1 X 2 , X3 ,..., X n  x1 , x2 ,..., xn 
X 2 X3
Xn
n -1 summations
Joint PDF of n random variables
If X is a continuous random vector, that is, FX1 , X 2 ,.. X n ( x1 , x2 ,..xn ) is continuous in each of
its arguments, then X can be specified by the joint probability density function
f X (x)  f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )

n
FX , X ,.. X ( x1 , x2 ,..xn )
x1x2 ...xn 1 2 n
Properties of joint PDF of n random variables
The joint pdf of n random variables satisfies the following important properties
(1) f X1 , X 2 ,.. X n ( x1 , x2 ,..xn ) is always a non-negative quantity. That is,
f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )  0 ( x1 , x2 ,..xn ) 
 

 

n
(2)   ...  f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )dx1dx2 ...dxn  1
(3) Given
f X (x)  f X1 , X 2 ,.. X n ( x1 , x2 ,..xn ) for all ( x1 , x2 ,..xn )  R n , we can find the
probability of a Borel set (region ) B  R n ,
P({( x1 , x2 ,..xn )  B})    ..  f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )dx1dx2 ...dxn
B
(4) The marginal CDF of a random variable X i is related to f X1 , X 2 ,.. X n ( x1 , x2 ,..xn ) by
the (n  1)  fold integral
 

 

f X i ( xi )    ...  f X1 , X 2 ,.. X n ( x1 , x2 ,..xi ..xn )dx1dx2 ...dxn
2
where the integral is performed over all the arguments except xi .
Similarly,
 

 

f X i , X j ( xi , x j )    ...  f X1 , X 2 ,.. X n ( x1 , x2 ,..xi ..xn )dx1dx2 ...dxn
( n  2)  fold integral
and so on.
The conditional density functions are defined in a similar manner. Thus
f X , X ,..., X  x1 , x2 ,........., xn 
f X m 1 , X m  2 ,......, X n / X1 , X 2 ,......., X m  xm1 , xm2 ,...... xn / x1 , x2 ,........., xm   1 2 n
f X1 , X 2 ,..., X m  x1 , x2 ,........., xm 
Independent random variables
The random variables X1 , X 2 ,..............., X n are called (mutually) independent if and only
if
n
f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )   f X i  xi 
i 1
For example, if X1 , X 2 ,..............., X n are independent Gaussian random variables, then
n
f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )  
i 1
1
e
2 i

1  xi  i 
2  i2
2
Remark, X1 , X 2 ,................, X n may be pair wise independent, but may not be mutually
independent.
Identically distributed random variables:
The random variables X1 , X 2 ,....................., X n are called identically distributed if each
random variable has the same marginal distribution function, that is,
FX1  x   FX 2  x   .........  FX n  x 
x
An important subclass of independent random variables is the independent and identically
distributed (iid) random variables. The random variables X1 , X 2 ,....., X n are called iid if
X1 , X 2 ,......., X n are mutually independent and each of X1 , X 2 ,......., X n has the same
marginal distribution function.
Example: If X1 , X 2 ,......., X n may be iid random variables generated by n independent
throwing of a fair coin and each taking values 0 and 1, then X1 , X 2 ,......., X n are iid and
1
p X 1,1,......,1   
2
n
3
Moments of Multiple random variables
Consider
n
jointly
random
variables
represented
by
the
random
vector
X  [ X 1 , X 2 ,....., X n ]'. The expected value of any scalar-valued function g ( X) is defined
using the n  fold integral as
 

 

E ( g ( X)    ...  g ( x1 , x2 ,..xn ) f X1 , X 2 ,.. X n ( x1 , x2 ,..xn )dx1dx2 ...dxn
The mean vector of X, denoted by μ X , is defined as
μ X  E ( X)
  E ( X 1 ) E ( X 2 )......E ( X n )  '.
   X1  X 2 ...  X n  '.
Similarly for each (i, j ) i  1, 2,.., n, j  1, 2,.., n we can define the covariance
Cov( X i , X j )  E( X i   Xi )( X j   X j )
All the possible covariances can be represented in terms of a matrix called the covariance
matrix CX defined by
CX  E ( X  X )( X  X )
cov( X 1 , X 2 ) cov( X 1 , X n ) 
 var( X 1 )
cov( X , X ) var( X ) . cov( X , X ) 
2
1
2
2
n 





var( X n ) 
 cov( X n , X 1 ) cov( X n , X 2 )
Properties of the Covariance Matrix

CX is a symmetric matrix because Cov( X i , X j )  Cov( X j , X i )

CX is a non-negative definite matrix in the sense that for any real vector Z  0,
the quadratic form z CX z  0. The result can be proved as follows:
z CX z  z E (( X   X )( X   X ))z
 E (z ( X   X )( X   X )z )
 E (z ( X   X )) 2
0
Remark
 Non-negative definiteness of the matrix CX implies that all its eigen values
are nonnegative. All the leading minors at the top-left corner of a nonnegative definite matrix are non-negative. This property can be used to
check the non-negative definiteness of a matrix in the simple case.
 The covariance matrix represents second-order relationship between each
pair of the random variables and plays an important role in applications of
random variables.
4
Example
2 1
The symmetric matrix 1 2

1 1
2 1
2 1 
det 
 3 and det 1 2


1 2 
1 1
1
1 is non-negative definite because
2 
1
1 =4.
2 
2 1 3
The symmetric matrix 1 2 1  is non-negative definite because


3 1 2 
 2 1 3
det 1 2 1   8  0
3 1 2
Uncorrelated random variables
n random variables X1 , X 2 ,.., X n are called uncorrelated if for each
(i, j ) i  1, 2,.., n, j  1, 2,.., n
Cov( X i , X j )  0
If
X1 , X 2 ,.., X n are uncorrelated, CX will be a diagonal matrix.
Multiple Jointly Gaussian Random variables
For any positive integer n, X1 , X 2 ,....., X n represent n jointly random variables. These n
random variables define a random vector X  [ X1 , X 2 ,....., X n ]'. These random variables are
called jointly Gaussian if the random variables X 1 , X 2 ,....., X n have joint probability
density function given by
f X1 , X 2 ,....., X n ( x1 , x2 ,...xn ) 
1
 ( X μ X )' CX1 ( X μ X )
e 2

2
CX  E ( X  μ X )( X  μ X )'
Where

n
det(CX )
is
the
covariance
matrix
and
μ X  E ( X)   E ( X 1 ), E ( X 2 )......E ( X n )  ' is the vector formed by the means of the random
variables.
Remark



The properties of the two-dimensional Gaussian random variables can be
extended to multiple jointly Gaussian random variables.
If X1 , X 2 ,....., X n are jointly Gaussian, then the marginal PDF of each of
X1 , X 2 ,....., X n is a Gaussian.
If the jointly Gaussian random variables X1 , X 2 ,....., X n are uncorrelated, then
X1 , X 2 ,....., X n are independent also.
5
Proof
CX is a diagonal matrix given by
 12

0
CX  
...

0
0
 22
...
0
... 0 

... 0 


... n2 
Then
 1
 2
 1

0
-1
CX  
...

0


... 0 

1

... 0 
2
2
 and

...

1
0 ... 2 
 n 
0
det(CX )   12 22 ... n2
Therefore,
f X1 , X 2 ,....., X n ( x1 , x2 ,...xn ) 

1
 ( X μ X )' CX1 ( X μ X )
e 2


n

i1

2
e
det(CX )
1 n  xi   X i
  
2 i 1  i
2
e
n

n



2
 12 22 ... n2
1  xi   X i
 
2 i



2
2 i2
Thus X1 , X 2 ,....., X n are independent also.
6