Download Bayesian Prior and Posterior

Bayesian Prior and
Posterior
Study Guide for ES205
Yu-Chi Ho
Jonathan T. Lee
Nov. 24, 2000
Outline
Conditional Density
 Bayes Rule
 Conjugate Distribution
 Example
 Other Conjugate Distributions
 Application

2
Conditional Density

The conditional probability density of
w happening given x has occurred,
assume px(x)  0:
pW , X w, x 
pW | X w | x  
p X x 
pW , X w, x   pW | X w | x  p x x 
 p X |W x | w p w w
N
3
Bayes Rule

Replace the joint probability density
function with the bottom equation
from page 3:
pW | X 1 ,, X n w | x1 , xn 

N
p X 1 ,, X n |W x1 ,, xn | w pW w
p X 1 ,, X n x1 ,, xn 
4
Conjugate Distribution
W: parameter of interest in some
system
 X: the independent and identical
observation on the system
 Since we know the model of the
system, the conditional density of
X|W could be easily computed, e.g.,

p X |W x | w
N
5
Conjugate Distribution
(cont.)

N
If the prior distribution of W belong
to a family, for any size n and any
values of the observations in the
sample, the posterior distribution of
W must also belong to the same
family. This family is called a
conjugate family of distributions.
6
Example
An urn of white and red balls with
unknown w being the fraction of the
balls that are red.
 Assume we can take n sample, X1, …,
Xn, from the urn, with replacement,
e.g, n i.i.d. samples.
This is a Bernoulli distribution.

N
7
Example (cont.)

Total number of red ball out of n
trials, Y = X1 + … + Xn, has the
binomial distribution
p X1 ,

, X n |W
 x1 ,
, xn | w   w 1  w 
y
Assume the prior dist. of w is beta
distribution with parameters  and 
pW W   w
 1
N
n y
1  w
 1
8
Example (cont.)

The posterior distribution of W is
pW | X 1 ,, X n w | x1 , xn 
 p X 1 ,, X n |W x1 ,, xn | w pW w
  y 1
w
1  w
  n  y 1
which is also a beta distribution.
9
Example (cont.)

Updating formula:
•
•
N
’ =  + y
Posterior (new) parameter =
prior (old) parameter + # of red balls
’ =  + (n – y)
Posterior (new) parameter
= prior (old) parameter + # of white
balls
10
Other Conjugate
Distributions




N
The observations forms a Poisson
distribution with an unknown value of the
mean w.
The prior distribution of w is a gamma
distribution with parameters  and .
The posterior is also a ngamma distribution
with parameters   i 1 xi and  + n.
Updating formula:
’ =  + y
’ =  + n

11
Other Conjugate
Distributions (cont.)




N
The observations forms a negative binomial
distribution with a specified r value and an
unknown value of the mean w.
The prior distribution of w is a beta
distribution with parameters  and .
The posterior is also a beta distribution
n


with parameters  + rn and
i1 xi .
Updating formula:
’ =  + rn
’ =  + y
12
Other Conjugate
Distributions (cont.)




The observations forms a normal
distribution with an unknown value of the
mean w and specified precision r.
The prior distribution of w is a normal
distribution with mean  and precision .
The posterior is also a normal distribution
  nx r
with mean
and precision  + nr.
  nr
Updating formula:
 
N
  nx r
  nr
     nr
13
Other Conjugate
Distributions (cont.)




N
The observations forms a normal
distribution with the specified mean m
and unknown precision w.
The prior distribution of w is a gamma
distribution with parameters  and .
The posterior is also a gamma distribution
n
n
1




with parameters
and 2 i 1 xi  m2 .
2
Updating formula:
’ =  + n/2 n
’ =  + ½ i 1 xi  m2
14
Summary of the Conjugate
Distributions
N
Observations
Prior
Posterior
Bernoulli
Beta
Beta
Poisson
Gamma Gamma
Negative
binominal
Normal
Beta
Normal
Gamma Gamma
Beta
Normal Normal
15
Application

N
Estimate the state of the system
based on the observations: Kalman
filter.
16
References:
• DeGroot, M. H., Optimal Statistical Decisions,
McGraw-Hill, 1970.
• Ho, Y.-C., Lecture Notes, Harvard University, 1997.
• Larsen, R. J. and M. L. Marx, An Introduction to
Mathematical Statistics and Its Applications,
Prentice Hall, 1986.
17