Download Random Variables and Random Vectors

Random Variables
and
Random Vectors
Tim Marks
University of California San Diego
Tim Marks, Dept. of Computer Science and Engineering
Good Review Materials
http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm
• (Gonzales & Woods review materials)
• Chapt. 1: Linear Algebra Review
• Chapt. 2: Probability, Random Variables, Random
Vectors
Tim Marks, Dept. of Computer Science and Engineering
Random variables
• Samples from a random variable are real numbers
– A random variable is associated with a probability distribution
over these real values
– Two types of random variables
• Discrete
– Only finitely many possible values for the random variable:
X  {a1, a2, …, an}
– (Could also have a countable infinity of possible values)
» e.g., the random variable could take any positive integer value
– Each possible value has a finite probability of occurring.
• Continuous
– Infinitely many possible values for the random variable
– E.g., X  {Real numbers}
Tim Marks, Dept. of Computer Science and Engineering
Discrete random variables
• Discrete random variables have
a pmf (probability mass function), P
P(X = a) = P(a)
• Example: Coin flip
1
X = 0 if heads
X = 1 if tails
– What is the pmf of this
random variable?
P (a )
0.5
0
0
Tim Marks, Dept. of Computer Science and Engineering
1
a
Discrete random variables
• Discrete random variables have
a pmf (probability mass function), P
1
P(X = a) = P(a)
P (a )
• Example: Die roll
X  {1, 2, 3, 4, 5, 6}
– What is the
pmf of this
random variable?
0.5
0
1
2
Tim Marks, Dept. of Computer Science and Engineering
3
4
5
6
a
Continuous random variables
• Continuous random variables have a
pdf (probability density function), p
• Example: Uniform distribution
p(x)
p(1.3) = ?
0.5
What is the probability
that X = 1.3 exactly:
P(X = 1.3) = ?
0
1
2
x
p(2.4) = ?
Probability corresponds to area
under the pdf.
1.5
P(1 < X < 1.5) = ? p(x) dx  0.25
1
Tim Marks, Dept. of Computer Science and Engineering
Continuous random variables
• What is the total area under any pdf ?

 p( x)dx  1?

• Example continuous random variable:
Human heights
p(h)
p(h)
h
p(h)
h
Tim Marks, Dept. of Computer Science and Engineering
h
Random variables
• How much change do you have on you?
– Asking a person (chosen at random) that question can
be thought of as sampling from a random variable.
• Is the random variable
“Amount of change people carry”
discrete or continuous?
Tim Marks, Dept. of Computer Science and Engineering
Random variables: Mean & Variance
• These formulas can be used to find the mean and
variance of a random variable when its true
probability distribution is known.
Definition
Mean

Variance

Var(X)

  E( X)
Discrete r.v.
   ai P(ai )
Continuous r.v.

E(X  )2 
 a  
i
i

P(ai )

Tim Marks, Dept. of Computer Science and Engineering

 x p(x) dx

i
2


 (x  )

2
p(x) dx
An important type of random variable
Tim Marks, Dept. of Computer Science and Engineering
The Gaussian distribution
1
p(x) 
2 

e
(x  )2
2
2
X ~ N,  2)

Tim Marks, Dept. of Computer Science and Engineering
Estimating the Mean & Variance
– After sampling from a random variable n times,
these formulas can be used to estimate the
mean and variance of the random variable.
• Samples x1, x2, x3, …, xn
Estimated mean:
Estimated variance:
1 n
m   xi
n i1
n
1
2
2
   x i  m
n i1
 maximum
likelihood estimate
n
1
2
2
 
x i  m  unbiased estimate

n 1 i1
Tim Marks, Dept. of Computer Science and Engineering
Finding mean, variance in Matlab
xn ]
– Samples x  [x1 x2 x3
– Mean
>> m = (1/n)*sum(x)

– Variance
2 

1
[x1  m
n
Method 1:
Method 2:
x2  m
x1  m 


x

m
2

x n  m] 




x

m
 n

>> v = (1/n)*(x-m)*(x-m)’
>> z = x-m
>> v = (1/n)*z*z’
Tim Marks, Dept. of Computer Science and Engineering
Example continuous random variable
• People’s heights (made up)
– Gaussian
 = 67 , 2 = 20
• What if you went
to a planet where
heights were
Gaussian with
 = 75 , 2 = 5
– How would they be
different from us?
Tim Marks, Dept. of Computer Science and Engineering
Example continuous random variable
• Time people woke up this morning
– Gaussian
 = 9 , 2 = 1
Tim Marks, Dept. of Computer Science and Engineering
Random vectors
– An n-dimensional random vector consists of n random
variables all associated with the same probability space.
• Each outcome dictates the value of every random variable.
– Example 2-D random vector:
X  where X is random variable of human heights
V   
Y 
Y is random variable of wake-up times
– Sample n times from V.

v1 v 2
x1 x 2

y1 y 2
vn
x n 

y n 
Let’s collect some
samples and graph them:
y
(wake-up
times)
x (heights)
Tim Marks, Dept. of Computer Science and Engineering
Random Vectors
• The probability distribution of the random vector P(V) is
P(X, Y) , the joint probability distribution of the random
variables X and Y.
• What will the graph of samples from V look like?
• Find m, the mean of V
– (Mean of X is 67)
– (Mean of Y is 10)
67
m   
10
Tim Marks, Dept. of Computer Science and Engineering
Mean of a random vector
• Estimating the mean
of a random vector
– n samples from V
Mean
v1 v 2
x1 x 2

y1 y 2
1 n  1 n x i  mx 
m   v i     
n i1
n i1 y i  my 
– To estimate mean of V in Matlab
>>
 (1/n)*sum(v,2)
Tim Marks, Dept. of Computer Science and Engineering
vn
x n 

y n 
Random vector
– Example 2-D random vector:
X 
V   
Y 
where X is random variable of human heights
Y is random variable of human weights
• Sample n times from V
– What will graph

look like?
v1 v 2
x1 x 2

y1 y 2

Tim Marks, Dept. of Computer Science and Engineering
vn
x n 

y n 

Covariance of two random variables
• Height and wake-up time are uncorrelated, but
height and weight are correlated.
• Covariance
Cov(X, Y) = 0 for X = height, Y = wake-up times
Cov(X, Y) > 0 for X = height, Y = weight
– Definition:
Cov(X,Y)  E(X  x )(Y  y )
If Cov(X, Y) < 0 for two random variables X, Y , what would a
scatterplot of samples from X, Y look like?
Tim Marks, Dept. of Computer Science and Engineering
Estimating covariance from samples
• Sample n times:
x1

y1
x2
y2
x n 

y n 
n
1
Cov(X,Y )   (x i  mx )(y i  my )
n i1

n
1
Cov(X,Y ) 
(x i  mx )(y i  my )

n 1 i1
 maximum
likelihood estimate
 unbiased estimate
• Cov(X, X) = ?Var(X)
• How are Cov(X, Y) and Cov(Y, X) related?
Cov(X, Y) = Cov(Y, X)
Tim Marks, Dept. of Computer Science and Engineering



Estimating covariance in Matlab
– Samples
x  [x1 x2 x3
y  [y1 y2 y3
– Means
mx  m_x
my  m_y
xn ]
yn ]
– Covariance
1
Cov( X,Y)  [x1  mx
n
x 2  mx
y1  my 


y

m
2
y 
x n  mx ] 




y n  my 
Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)’
Method 2: >> w = x-m_x
>> z = y-m_y
>> v = (1/n)*w*z’
Tim Marks, Dept. of Computer Science and Engineering
Covariance matrix of a
D-dimensional random vector
• In 2 dimensions
X 
V   
Y 
T
Cov(V)  E V   V   

X  X 
 E
X  X
Y  Y 

  Var( X) Cov( X,Y)
Y  Y  

Cov(
X,Y
)
Var(Y
)

 
• In D dimensions
T
Cov(V)  E V  V  
• When is a covariance matrix symmetric?
A. always, B. sometimes, or C. never

Tim Marks, Dept. of Computer Science and Engineering
Example covariance matrix
• People’s heights
(made up)
X ~ N(67, 20)
• Time people woke up
this morning
Y ~ N(9, 1)
• What is the covariance
matrix of
X 
V   
Y 
?
Tim Marks, Dept. of Computer Science and Engineering
20 0


0 1

Estimating the covariance matrix from
samples (including Matlab code)
– Sample n times and find mean of samples
v1 v 2
x1 x 2
V  
y1 y 2
vn
x n 

y n 
– Find the covariance matrix

1
Cov(V) 
n
x1  mx

y1  my
x 2  mx

y 2  my
m x 
m   
m y 
x1  mx

x n  mx  x 2  mx

y n  my  

x n  mx
>> m = (1/n)*sum(v,2)
>> z = v - repmat(m,1,n)
>> v = (1/n)*z*z’
Tim Marks, Dept. of Computer Science and Engineering
y1  my 

y 2  my 


y n  my 
Gaussian distribution in D dimensions
• 1-dimensional Gaussian is completely determined
by its mean, , and variance,  2 :
X ~ N, 
2)
1
p(x) 
2 

e
(x  )2
2
2
• D-dimensional Gaussian is completely determined
by its mean, , and covariance matrix,  :
X ~ N,)
p(x) 
1
(2 )
D /2
 12 (x )T  1 (x )

1/ 2
e
–What happens when D = 1 in the Gaussian formula?
Tim Marks, Dept. of Computer Science and Engineering