Download docx

Bernoulli Distribution
𝑓(1) = 𝑝(𝑋 = 1) = 𝜃
𝑓(0) = 𝑝(𝑋 = 0) = 1 − 𝜃
𝑝(𝑥|𝜃) = 𝜃 𝑥 (1 − 𝜃)1−𝑥
Multinomial Distriubtion
𝑓(1) = 𝑝(𝑋 = 1) = 𝜃1
𝑓(𝑘) = 𝑝(𝑋 = 𝑘) = 𝜃𝑘
𝑘
∑ 𝜃𝑐 = 1
𝑐=1
Maximum Likelihood
Assume data is independent, identically distributed
𝑛
𝑝(𝑥1 , 𝑥2 , … , 𝑥𝑛 |𝜃) = ∏ 𝑝(𝑥𝑖 |𝜃)
𝑖=1
Find 𝜃 that maximizes the joint probability
If given a function, the minima / maxima will occur when the derivative is at zero. It is necessary to take
the second derivative to ensure that you have found a maxima (should be negative)
Contiguous Probability Distributions
Probability Density Function
𝑏
𝑃(𝑎 < 𝑥 < 𝑏) = ∫ 𝑝(𝑥)𝑑𝑥
𝑤ℎ𝑒𝑟𝑒 𝑝(𝑥)𝑖𝑠 𝑎 𝑑𝑒𝑛𝑠𝑖𝑡𝑦 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛
𝑎
∞
∫ 𝑝(𝑥)𝑑𝑥 = 1
−∞
∞
𝐸(𝑥) = ∫ 𝑥𝑝(𝑥)𝑑𝑥
−∞
Sum Rule:
𝑝(𝑥) =
∞
∫−∞ 𝑝(𝑥, 𝑦)𝑑𝑥
Uniform Distribution
1
𝑈𝑛𝑖𝑓(𝑎, 𝑏) = {𝑏 − 𝑎 𝑎 ≤ 𝑥 ≤ 𝑏
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Things are only possible between a and b
∞
𝑏
𝐸(𝑥) − ∫ 𝑥𝑝(𝑥)𝑑𝑥 = ∫ 𝑥
−∞
𝑎
𝑏
1
1
1 𝑥2
𝑑𝑥 =
∫ 𝑥𝑑𝑥 =
𝑏−𝑎
𝑏−𝑎 𝑎
𝑏−𝑎 2
Gaussian Distribution
𝑝(𝑥|𝜇, 𝜎 2 ) =
1
√2𝜋𝜎 2
∗ exp (−
∞
√2𝜋𝜎 2 = ∫ exp (−
−∞
1
(𝑥 − 𝜇)2 )
2𝜎 2
1
(𝑥 − 𝜇)2 ) 𝑑𝑥
2𝜎 2
Maximum Likelihood in a Gaussian Distribution
𝑥1 , 𝑥2 , … , 𝑥𝑛 : IID Samples
𝑛
𝑝(𝑥1 , 𝑥2 , … , 𝑥𝑛 |𝜇, 𝜎
2)
= ∏ 𝑝(𝑥𝑖 |𝜇, 𝜎 2 )
𝑖=1
𝑛
= ∏
𝑖=1
𝑖
1
exp (− 2 (𝑥𝑖 − 𝜇)2 )
2
.5
(2𝜋𝜎 )
2𝜎
𝑛
𝑖
1
=
exp (− 2 ∑
2
.5
(2𝜋𝜎 )
2𝜎
(𝑥𝑖 − 𝜇)2 )
𝑖=1
Take Log
𝑛
𝑛
𝑛
1
− ln(2𝜋) − ln(𝜎 2 ) = 2 ∑(𝑥𝑖 − 𝜇)2
2
2
2𝜎
𝑖=1
Take derivative with 𝜇
𝑛
1
∑(𝑥𝑖 − 𝜇) = 0
𝜎2
𝑖=1
𝑛
𝜇𝑚𝑙
1
= ∑ 𝑥𝑖
𝑛
𝑖=1
For 𝜎 2
(𝑠𝑎𝑚𝑝𝑙𝑒 𝑚𝑒𝑎𝑛)
𝑏
|
𝑎
𝑛
2
𝜎𝑀𝐿
1
= ∑(𝑥𝑖 − 𝜇𝑀𝐿 )2
𝑛
(𝑠𝑎𝑚𝑝𝑙𝑒 𝑣𝑎𝑟𝑖𝑒𝑛𝑐𝑒)
𝑖=1
Multivariate Gaussian Distribution
More than one random variable
Represent the set of random variables in a vector
Now let 𝑥⃗ be a vector.
𝑢
⃗⃗ = 𝑣𝑒𝑐𝑡𝑜𝑟 𝑜𝑓 𝑙𝑒𝑛𝑔𝑡ℎ 𝑑
𝑝(𝑥⃗|𝑢
⃗⃗, ∑) =
1
𝑑
1
(2𝜋) 2 |∑|2
−1
1
exp(− (𝑥 − 𝜇)𝑇 ∑ ∑ (𝑥 − 𝜇)
2
Suppose ∑ = 𝜎 2 , 𝐼
2
∑ = (𝜎
0
0)
𝜎2
If the covariance is, zero, then 𝑥1 , 𝑥2 are independent variables
Properties
1. Suppose 𝑥⃗ is a multivariate Gaussian
𝑥⃗~𝑁(𝑢
⃗⃗, ∑)
(𝑜𝑟 𝑁(𝑥⃗|𝑢
⃗⃗, ∑) )
Partition 𝑥⃗:
𝑥⃗
𝑥⃗ = ( a )
𝑥⃗b
𝑊ℎ𝑎𝑡 𝑖𝑠 𝑝(𝑥⃗𝑎 |𝑥
⃗⃗⃗⃗⃗)?
Multivariate Gaussian.
𝑏
2. Sum Rule
𝑝(𝑥⃗𝑎 ) = ∫ 𝑝(𝑥⃗)𝑑𝑥⃗𝑏
Also a multivariate Gaussian
3. Bayes Rule for Gaussians
𝑝(𝑥|𝑦) =
𝑝(𝑦|𝑥)𝑝(𝑥)
𝑝(𝑦)
If 𝑝(𝑦|𝑥) and 𝑝(𝑥) are Gaussian, then 𝑝(𝑦) and 𝑝(𝑥|𝑦) are Gaussian