Download Random Vectors and Matrices

Random Vectors and Matrices
• A random vector is a vector whose elements are random variables.
• The collective behavior of a p x 1 random vector is described by a
joint probability density function f(x1,x2,…,xp) = f(x).
• If the joint density of a p x 1 random vector can be factored as
f(x1,x2,…,xp) = f1(x1) f2(x2)∙∙∙ fp(xp) then the p continuous random
variables X1,X2,…Xp are mutually independent.
STA347 - week 9
1
Mean and Variance of Random Vector
• The expected value of a random vector is a vector of the expected values of
each of its elements. That is, the population mean vector is
• The population variance-covariance matrix of a px1 random vector x is a
p x p symmetric matrix
  CovX   E x   x   '   ij 
where σij = Cov(Xi, Xj) = E(Xi – μi)(Xj – μj).
• The population correlation matrix of a px1 random vector x is a p x p
symmetric matrix ρ = (ρij)
where  ij 
 ij
 ii jj
.
STA347 - week 9
2
Properties of Mean Vector and Covariance Matrix
STA347 - week 9
3
Functions of Random variables
• In some case we would like to find the distribution of Y = h(X) when the
distribution of X is known.
• Discrete case

  PX  x
 
pY  y   PY  y   Ph X   y   P X  h 1  y  
xh 1 y
• Examples
1. Let Y = aX + b , a ≠ 0
1


PY  y   PaX  b  y   P X   y  b 
a


2. Let Y
 X2

 
P X  y  P X   y

PY  y   P X 2  y   P X  0
 0



STA347 - week 9

if y  0
if y  0
if y  0
4
Continuous case – Examples
1. Suppose X ~ Uniform(0, 1). Let Y  X 2 , then the cdf of Y can be found as
follows
FY  y   PY  y   P X 2  y  P X  y  FX y
 


 
The density of Y is then given by
2. Let X have the exponential distribution with parameter λ. Find the
1
density for Y 
X 1
3. Suppose X is a random variable with density
x 1

f X x    2
 0
, 1  x  1
, elsewhere
Check if this is a valid density and find the density of Y  X 2 .
STA347 - week 9
5
Theorem
• If X is a continuous random variable with density fX(x) and h is strictly
increasing and differentiable function form R  R then Y = h(X) has density
fY  y  f X

 

d 1
h y
h y
dy
1
for y  R .
• Proof:
STA347 - week 9
6
Theorem
• If X is a continuous random variable with density fX(x) and h is strictly
decreasing and differentiable function form R  R then Y = h(X) has density

 dyd h  y 
f Y  y    f X h 1  y 
1
for y  R .
• Proof:
STA347 - week 9
7
Summary
• If Y = h(X) and h is monotone then
d 1
f Y  y   f X h  y  h  y 
dy
1
• Example
X has a density
 x3

f X x    4
0

for 0  x  2
otherwise
Let Y  X 6. Compute the density of Y.
STA347 - week 9
8
Change-of-Variable for Joint Distributions
• Theorem
Let X and Y be jointly continuous random variables with joint density
function fX,Y(x,y) and let DXY = {(x,y): fX,Y(x,y) >0}. If the mapping T given
by T(x,y) = (u(x,y),v(x,y)) maps DXY onto DUV. Then U, V are jointly
continuous random variable with joint density function given by
 f xu , v , y u , v  J u , v 
fU ,V u , v    X ,Y
0

if u, v  DU ,V
otherwise
where J(u,v) is the Jacobian of T-1 given by
x
J u, v   u
y
u
x
v
y
v
assuming derivatives exists and are continuous at all points in DUV .
STA347 - week 9
9
Example
• Let X, Y have joint density function given by
e  x y
f X ,Y x, y   
 0
Find the density function of U 
if x, y  0
otherwise
X
.
X Y
STA347 - week 9
10
Example
• Show that the integral over the Standard Normal distribution is 1.
STA347 - week 9
11
Example
• A device containing two key components fails when and only when both
components fail. The lifetime, T1 and T2, of these components are
independent with a common density function given by
 e t
fT t   
 0
t 0
otherwise
• The cost, X, of operating the device until failure is 2T1 + T2. Find the
density function of X.
STA347 - week 9
12
Convolution
• Suppose X, Y jointly distributed random variables. We want to find the
probability / density function of Z=X+Y.
• Discrete case
X, Y have joint probability function pX,Y(x,y). Z = z whenever X = x and
Y = z – x. So the probability that Z = z is the sum over all x of these joint
probabilities. That is
pZ z    p X ,Y x, z  x .
x
• If X, Y independent then
pZ z    p X x  pY z  x .
x
This is known as the convolution of pX(x) and pY(y).
STA347 - week 9
13
Example
• Suppose X~ Poisson(λ1) independent of Y~ Poisson(λ2). Find the
distribution of X+Y.
STA347 - week 9
14
Convolution - Continuous case
• Suppose X, Y random variables with joint density function fX,Y(x,y). We want to
find the density function of Z=X+Y.
Can find distribution function of Z and differentiate. How?
The Cdf of Z can be found as follows:

FZ  z   P X  Y  z  
zx
  f x, y dydx
X ,Y
x  y 

z
  f x, v  x dvdx

X ,Y
x  v 
z


  f x, v  x dxdv.
X ,Y
v  x 

If
 f x, v  x dx
XY
is continuous at z then the density function of Z is given by
x 
• If X, Y independent then
f Z z  
f Z z  

 f x, z  xdx
XY
x 

 f x f z  xdx
X
Y
x
This is known as the convolution of fX(x) and fY(y).
15
Example
• X, Y independent each having Exponential distribution with mean 1/λ. Find
the density for W=X+Y.
STA347 - week 9
16
Order Statistics
• The order statistics of a set of random variables X1, X2,…, Xn are the same
random variables arranged in increasing order.
• Denote by
X(1) = smallest of X1, X2,…, Xn
X(2) = 2nd smallest of X1, X2,…, Xn

X(n) = largest of X1, X2,…, Xn
• Note, even if Xi’s are independent, X(i)’s can not be independent since
X(1) ≤ X(2) ≤ … ≤ X(n)
• Distribution of Xi’s and X(i)’s are NOT the same.
STA347 - week 9
17
Distribution of the Largest order statistic X(n)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The CDF of the largest order statistic, X(n), is given by
FX  n  x   PX n   x 
• The density function of X(n) is then
f X  n  x  
d
FX  n   x  
dx
STA347 - week 9
18
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(n).
STA347 - week 9
19
Distribution of the Smallest order statistic X(1)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The CDF of the smallest order statistic X(1) is given by
FX 1 x  PX 1  x  1  PX 1  x 
• The density function of X(1) is then
f X 1 x  
d
FX x  
dx 1
STA347 - week 9
20
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(1).
STA347 - week 9
21
Distribution of the kth order statistic X(k)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The density function of X(k) is
f X  k  x  
n!
FX x k 1 1  FX x nk f X x 
k  1!n  k !
STA347 - week 9
22
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(k).
STA347 - week 9
23
Computer Simulations - Introduction
• Modern high-speed computers can be used to perform simulation
studies.
• Computer simulation methods are commonly used in statistical
applications; sometimes they replace theory, e.g., bootstrap methods.
• Computer simulations are becoming more and more common in
many applications such as quality control, marketing, scientific
research etc.
STA347 - week 9
24
Applications of Computer Simulations
• Our main focus is on probabilistic simulations. Examples of
applications of such simulations include:
 Simulate probabilities and random variables numerically.
 Approximate quantities that are too difficult to compute
mathematically.
 Random selection of a sample from a very large data sets.
 Encrypt data or generate passwords.
 Generate potential solutions for difficult problems.
STA347 - week 9
25
Steps in Probabilistic Simulations
• In most applications, the first step is to specify a certain probability
distribution.
• Once such distribution is specified, it will be desired to generate one
or more random variables having that distribution.
• The build-in computer device that generates random numbers is
called pseudorandom number generator.
• It is a device for generating a sequence U1, U2, … of random values
that are approximately independent and have approximately uniform
distribution of the unit interval [0,1].
STA347 - week 9
26
Simulating Discrete Distributions - Example
• Suppose we wish to generate X ~ Bernoulli(p), where 0 < p < 1.
• We start by generating U ~ Uniform[0, 1] and then set:
1
X 
0
Up
Up
• Then clearly X takes two values, 0 and 1. Further,
P X  1  PU  p  p
• Therefore, we have that X ~ Bernoulli(p).
• This can be generalized to generate Y ~ Binomial(n, p) by generating
U1, U2, … Un. Setting Xi as above and let Y = X1 + ∙∙∙ + Xn.
STA347 - week 9
27
Simulating Discrete Distributions
• In general, suppose we wish to generate a random variable with
probability mass function p.
• Let, x1 < x2 < x3 < ∙∙∙ be all the values for which p(xi) > 0.
• Let U ~ Uniform[0, 1].
• Define Y by:
j


Y  min  x j :  pxk   U 
k 1


• Theorem 1: Y is a discrete random variable, having probability
mass function p.
• Proof:
STA347 - week 9
28
Simulating Continuous Distributions - Example
• Suppose we wish to generate X ~ Uniform[a, b].
• We start by generating U ~ Uniform[0, 1] and then set:
X  b  aU  a
• Using one-dimensional change of variable theorem we can easily
show that X ~ Uniform[a, b].
STA347 - week 9
29
Simulating Continuous Distributions
• In general, simulating continuous distribution is not an easy task.
• However, for certain continuous distributions it is not difficult.
• The general method for simulating continuous distribution makes
use of the inverse cumulative distribution function.
• The inverse cdf of a random variable X with cumulative distribution
function F is defined by:
F 1 t   min x : F x   t
for 0 < t < 1.
STA347 - week 9
30
Inversion Method for Generating RV
• Let F be any cumulative distribution function, and let
U ~ Uniform[0, 1].
• Define a random variable Y by: Y  F 1 U 
• Theorem 2: Y has cumulative distribution function given by F. That is,
PY  y   F  y 
• Proof:
STA347 - week 9
31
Important Notes
• The theorem above is valid for any cumulative distribution function
whether it corresponds to a continuous distribution, a discrete
distribution or a mixture of the two.
• The inversion method for generating random variables described
above can be used whenever the distribution function is not too
complicated and has a close form.
• For distributions that are too complicated to sample using the
inversion method and for which there is no simple trick , it may
still be possible to generate samples using Markov chain methods.
STA347 - week 9
32
Example – Exponential Distribution
• Suppose X ~ Exponential(λ). The probability density function of X is:
e x
f X x   
 0
• The cdf of X is:
x0
otherwise
x
F x    e  t dt  1  e  x
0
 x
• Setting U  F x   1  e
and solving for x we get…
• Therefore, by theorem 2 above, X   1 ln 1  U  where
U ~ Uniform[0, 1], has an Exponential(λ) distribution.
STA347 - week 9
33
Example – Standard Normal Distribution
• Suppose X ~ Normal(0,1). The cdf of X is denoted by Ф(x). It is
given by:
t2
x
1 2
 x   
e dt
 2
• Then, if U ~ Uniform[0, 1], by theorem 2 above
Y   1 U   min x : x   U 
has a N(0,1) distribution.
• However, since both Ф and Ф-1 don’t have a close form, i.e., it is
difficult to compute them, the inversion method for generating RV is
not practical.
STA347 - week 9
34