Download Parameter Estimation

Parameter Estimation
Based on what we know, we can choose different approaches for parameter estimation.
•
•
•
Bayesian approach: given likelihood, the a priori distribution, and cost function.
Maximum likelihood approach: given likelihood; assume a uniform distribution for the parameter.
Minimum variance unbiased estimate: given likelihood and cost function; no information for
prior.
Bayesian approach
Assume the parameter is random: ~Θ. And it has a distribution of
The conditional distribution of Y given Θ
The goal is to find a function
observation
.
is
. (Prior)
. (Likelihood)
such that it is the best guess of the true value of
To define “best”, we need a criterion, i.e. a cost function
estimators:
based on the
. Different cost functions give different
1) MMSE (minimum mean square error)
Θ
arg min
Θ|
It is the conditional mean of Θ given
.
2) MMAE (minimum mean absolute error)
Θ
is any point such that
|
|
Θ
Θ
,
|
|
Θ
Θ
,
It is the conditional median of Θ given
.
3) MAP (maximum a posteriori probability)
Consider a uniform cost function,
0
∆
,
1
∆
For an estimator
the average posterior cost given
in this case is
Δ
,Θ
1
Θ
∆
|
1
Δ
So
should be chosen to maximize
|
arg max
It is the conditional mode of Θ given
| :
.
MVUE (minimum-variance unbiased estimate)
,
Consider the conditional risk function
Λ. Since we do not know the
over Θ, as we do in the
distribution of , we cannot get the Bayesian risk by taking the average of
MMSE case. Here, what we do is to put a restriction of unbiasedness on the estimate and minimize
for each
Λ.
1). Some definitions:
•
Unbiased Estimate
,
An estimate whose expected value equal the true parameter value:
Λ.
Interpretation: Within the restriction of unbiasedness,
becomes the variance of the estimate under
, thus the name minimum-variance unbiased estimate (MVUE).
•
Sufficiency
A function is a sufficient statistic for
;
Λ if the distribution of Y conditioned on
Λ. We may simply say that is sufficient for .
~ does not depend on for
contains all the information in Y that is useful for estimating .
Interpretation:
•
Minimal Sufficiency
A function is minimal sufficient for
;
Λ.
•
when
;
Λ if it is a function of every other sufficient statistic for
Completeness
;
The family
0
Λ is said to be complete if the condition
1 for all
Λ.
0 for all
Λ implies that
Some propositions:
•
The factorization Theorem
A statistic
for all
•
is sufficient for
Γ and
if and only if there are functions
and
such that
Λ.
The Rao-Blackwell Theorem
Suppose that
is an unbiased estimate of
and that is sufficient for . Define
by
|
Then
is also an unbiased estimate of an unbiased estimate of
. Furthermore,
1.
With equality if and only if
Note: proof applies Jensen’s inequality:
for a convex function .
2). Complete sufficient statistics and MVUE
when ~ . If
;
Λ is
Suppose that is sufficient for , and let denote the distribution of
complete, then is said to be a complete sufficient statistics. It can be proved that any unbiased estimator
that is a function of a complete sufficient statistic is unique and thus is an MVUE.
3). Procedure for seeking MVUE
First, find a complete sufficient statistic for
Second, find any unbiased estimator
Then
|
of
;
Λ.
.
is an MVUE of
.
The Information Inequality
is an estimate of the parameter in a family
where
;
Λ , then the information inequality holds:
is known as fisher’s information for estimating from Y.
log
For unbiased estimator, the information inequality reduces to the Cramer-Rao lower bound (CRLB):
1
MLE (Maximum-likelihood estimate)
In the absence of prior information about , we can assume it is uniformly distributed over Λ. Then MAP
reduces to maximum-likelihood estimate:
max
It can be calculated from the likelihood equation
log
|
0
Notes:
1) It can be proved that only solution to the likelihood equation can achieve the CRLB.
2) MLE will not necessarily achieve CRLB; MLE is not necessarily unbiased.
3) For i.i.d. observations, MLE will asymptotically reach unbiasedness and CRLB as the number of
observations
∞.
1. Comparison between MVUE and MLE
MUVE
Unbiased
No systematic way to find
May have to sacrifice MSE to achieve unbiasedness
MLE
Not always unbiased
For i.i.d., asymptotical unbiasedness and efficiency
Relatively easy to get
May achieve a lower MSE when being biased