Download estimation

Introduction to Signal Estimation
Outline

94/10/14
2
Scenario
 For physical considerations, we know that the voltage is
between –V and V volts. The measurement is corrupted by
noise which may be modeled as an independent additive
zero-mean Gaussian random variable n. The observed
variable is r . Thus,
r an
 The probability density governing the observation process is
given as
2
 r a 2 2
pa (r ) 
94/10/14
1
e
2 n


n

3
Estimation model
H0
Probabilistic Mapping
to observation space
Observation
space
Estimation
Rule
Decision
Decision rule
H1
Parameter
Space
 Parameter Space: the output of the source is a
 Probabilistic Mapping from parameter space to observation
space :the probability law that governs the effect of parameters on
the observation space.
 Observation space: a finite-dimensional space
 Estimation rule: A mapping of observation space into an estimate
94/10/14
4
Parameter Estimation Problem
 In the composite hypothesis-testing problem，a family of
distributions on the observation space, indexed by a
parameter or set of parameters, a binary decision is wanted
to make about the parameter。
 In the estimation problem, values of parameters want to be
determined as accurately as possible from the observation
enbodied。
 Estimation design philosophies are different due to
o the amount of prior information known about the parameter
o the performance criteria applied。 .
94/10/14
5
Basic approaches to the parameter estimation
 Bayesian estimation:
o assuming that parameters under estimation are to be a random
quantity related statistically to the observation。
 Nonrandom parameter estimation:
o the parameters under estimation are assumed to be unknown but
without being endowed with any probabilistic structure。
94/10/14
6
Bayesian Parameter Estimation
 Statement of problem:
o  : the parameter space and the parameter   
o Y :random variable observation space
o p ;  :denoting a distribution on the observation space  , and
mapping from  to 

ˆ :   
Finding a function ˆ :    s.t ˆ is the best guess of the true
value of  based on the observation Y=y。

94/10/14
Obser.
R.V.
y
estimate
ˆ y 
ˆ y 
7
Bayesian Parameter Estimation
 Performance evaluation:
o the solution of this problem depends on the criterion of goodness by
which we measure estimation performance --the Cost function
assignment。

C :     R C ˆ Y    is
the cost of estimating a true value 
as ˆ for in  and ˆ in 
o The conditional risk/cost averaged over Y for each   
 




R     E  C  Y       C   y   p  y  dy
 
  

 
o The Bayes risk: if we adopt the interpretation that the actual
parameter value  is the realization of a random variable  , the
Bayes risk/average risk is defined as
   


r     E R       R   w   d
 
     
94/10/14
8
Bayesian Parameter Estimation
o where w   is the prior for random variable  。the appropriate design
goal is to find an estimator minimizing r ˆ  ，and the estimator is
known as a Bayes estimate of 
o Actually，the conditional risk can be rewritten as
 
 



R  E C  Y      E C  Y      


 
 

o the Bayes risk can be formulated as
  
   
 

 

r    E R     E E C  Y     E C  Y   
 
 

  
 
  
o the Bayes risk can also be written as
  
   
 
 



 

r    E R     E C  Y     E E C  Y   Y    E C  Y   Y  y p  y  dy
 

    

  
 

  
94/10/14
9
Bayesian Parameter Estimation
o the Bayes risk can also be written as
  
   
 
 



 

r    E R     E C  Y     E E C  Y   Y    E C  Y   Y  y p  y  dy
 

    

  
 

  
• The equation suggests that for each y   , the Bayes estimate of  can
be found by minimizing the posterior cost given Y=y
E C ˆ  y   Y  y


o Assuming that  has a conditional density w  y  given Y=y for
each y  , then the posterior cost ,given Y=y, is given by
 




E C   y   Y  y    C   y    w  y d


 
  
o Deriving w  y 
• If we know p  y  ,priori
 and w  
w  / y  
p  y  w  
p  y  w  

py 
 p  y  w   d

o the performance of Bayesian estimation depends on a cost function
94/10/14


C   y   


w  y 
10
(MMSE) Minimum-Mean-Squared-Error Estimation
2

 

 
  R and E     C  ,         for   ,    R 2

 



o Measuring the performance of an estimator in terms of the squared
of the estimation error ˆ  
2
ˆ
o The corresponding Bayes risk is E    

2



• defined as the mean-squared-error(MMSE) estimator。
o The Bayes estimation is the Minimum-mean-squared-error(MMSE)
estimator。
o the posterior cost given Y=y under this condition is given by
2
2

 

  



E    y     Y  y   E    y   2  y    2 Y  y 




 

2
  




 E    y  Y  y   2E   y   Y  y   E 2 Y  y 




 
2



   y   2  y  E  Y  y   E 2 Y  y 


94/10/14
11
(MMSE) Minimum-Mean-Squared-Error Estimation

o the cost function is a quadratic form of   y ,and is a convex function。
o Therefore, it achieves its unique minimum at the point where its

derivative   y  with respective to is zero。

10 H 10 D
D10



2
  E ˆ  y    Y  y 

  2 ˆ y   2E  Y  y  0


  
 ˆ  y 

ˆMMSE  y   E  Y  y      w  y d

o the conditional mean of  given Y=y 。 The estimator is also
sometimes termed the conditional mean estimator-CME。
94/10/14
12
(MMSE) Minimum-Mean-Squared-Error Estimation
 Another derivation:
2
2


 




E    y     Y  y       y     w  y d





 




2
  E ˆ  y    Y  y 

  2 ˆ y   w  y d  0
 
  
 ˆ  y 



 ˆ  y   w  y d     w  y d


ˆ  y      w  y d  E  Y  y 

94/10/14
13
(MMAE) : Minimum-Mean-Absolute-Error Estimation
 Measuring the performance of an estimator in terms of the
absolute value of the estimation error, ˆ  
  
  R and E      C  ,       for


ˆ,    R
2
 The corresponding Bayes risk is E ˆ   ，which is defined as
the mean-absolute-error (MMAE)。
 The Bayes estimation is the Minimum-mean-absoluteerror(MMAE) estimator。
94/10/14
14
(MMAE) : Minimum-Mean-Absolute-Error Estimation
 From the definition





E    y    Y  y    P ˆ  y     x Y  y dx

 0


  P    ˆ  y    x Y  y dx

  P   ˆ  y   x Y  y dx
0

0
 By change variable t  x  ˆ  y  and t   x  ˆ  y  for the first and
the second integration, respectively,


E ˆ  y    Y  y   ˆ P    t Y  y dt  
94/10/14

 y 
ˆ y 

P    t Y  y dt
15
(MMAE) : Minimum-Mean-Absolute-Error Estimation
 Actually,the expression represents that it is a differentiable
function of ˆ  y ， thus, it can be shown that

 E ˆ  y    Y  y

 ˆ  y 

  P      y  Y  y   P   ˆ  y  Y  y







o The derivative is a non-decreasing function of ˆ  y 
o If ˆ  y  approaches -  ,its value approaches -1
o If ˆ  y  approaches  ,its value approaches 1
o E ˆ  y    Y  y achieves its minimum over ˆ  y  at the point where its
derivative changes sign
P    t Y  y   P    t Y  y  t  ˆABS  y 
and
P    t Y  y   P    t Y  y  t  ˆABS  y 
94/10/14
16
(MMAE) : Minimum-Mean-Absolute-Error Estimation
 the cost function can be also expressed as

 
ˆ y 



E    y    Y  y    ˆ     w  y  d  
ˆ   w  y  d
 y 





 E ˆ  y    Y  y

ˆ y 
   ˆ w  y  d  
w  y  d  0
 y 

ˆ
  y 



ˆ y 




 
w  y  d   ˆ
 y 

1
w  y  d 
2
o the minimum-mean-absolute-error estimator is to estimate the
median of the conditional density function of  ,Y=y。
o the MMAE is also termed as condition median estimator
o For a given density function of , if its mean and median are the
same, then,MMSE and MMAE coincide each other,i.e. they have the
same performance based on different criterion adopted。

94/10/14
17
MAP Maximum A posterior Probability Estimation
 Assuming the uniform cost function
0 if ˆ    

C ˆ,    
0
1 if ˆ    


1
0
ˆ y    ˆ y 
ˆ y   
 The average posterior cost, given Y=y, to estimate ˆ  y is
given by
 




E C   y  ,  Y  y   P    y      Y  y 

 



 1  P ˆ  y      Y  y

94/10/14

18
MAP Maximum A posterior Probability Estimation
 Consideration I:
o Assuming is a discrete random variable taking values in a finite set
  0
M 1and with  i   j   for i  j the average posterior
cost is given as
 


E C   y  ,  Y  y   1  P   ˆ Y  y

 
 
 1  w ˆ y

 

which suggests that to minimize the average posterior cost,
The Bayes estimate in this case is given for each y   by any value
of  which can maximizes the posterior probability w  y  over   ，
i.e. the Bayes estimate is the value of  that has the maximum a
posterior probability of occurring given Y=y
94/10/14
19
MAP Maximum A posterior Probability Estimation
 Consideration II:
o  is a continuous random variable with conditional density function
given Y=y。Thus, the posterior cost become
w  y 


ˆ y  
E C ˆ  y  ,  Y  y  1   ˆ
w  y  d
  y  
ˆy  y 

w

• which suggests that the average posterior cost is minimized over ˆ  y  by

maximizing the area under over the interval   y      y   。



• Actually, the area can be approximately maximized by choosing ˆ  y  to
be a point of maximum of w  y  。
• the value  of can be chosen as small as possible and smooth w  y ,
then we can obtain

ˆ y  
  
ˆ y 

w  y  d  2w  y   ˆ y 
• where ˆ  y  is chosen to the value of  maximizing w  y  over  。
94/10/14
20
MAP Maximum A posterior Probability Estimation
 The MAP estimator can be formulated as


1  E C ˆ y ,  Y  y


ˆMAP  arg max w  / y 


w y 

ˆ y    ˆ y ˆ y   
 
o The uniform cost criterion leads to the procedure for estimating 
as that value maximizing the a posteriori density w  y , which is
known as the maximum a posteriori probability (MAP) estimate and
is denoted by ˆMAP。
o It approximates the Bayes estimate for uniform cost with small 
94/10/14
21
MAP Maximum A posterior Probability Estimation
 The MAP estimates are often easier to compute than MMSE、
MMAE，or other estimates。
 A density achieves its maximum value is termed a mode of
the corresponding probability。Therefore, the MMSE、
MMAE、and MAP estimates are the mean、median、and
mode of the corresponding distribution, respectively。
94/10/14
22
Remarks -Modeling the estimation problem
 Given conditions:
o Conditional probability density function of Y given    p ,   
o Prior distribution for  w  
o Conditional probability density of given Y=y
w  / y  
p  y  w  
 p  y  w   d

p  y  w  
py 

o The MMSE estimator ˆMMSE
ˆMMSE  y   E  Y  y      w  y d
o
The MMAE estimator  MMAE / ˆABS

ˆ y 

94/10/14



w  y  d   ˆ
 y 
w  y  d 
1
2
23
Remarks -Modeling the estimation problem
 The MAP estimator ˆMAP


ˆMAP  arg max w  / y 

o For MAP estimator, it is not necessary to calculate p  y because the
unconditional probability density of y does not affect the
maximization over  。
o ˆMAPcan be found by maximizing p  y  w   over   
.
o For the logarithm is an increasing function, therefore, ˆMAP also
maximizes log p  y   logw  
over    。
o If  is a continuous random variable given Y=y， then for sufficient
smooth p  y  and w   ,a necessary condition for MAP is given by
MAP equation


log
p
y


log w    



  
  MAP  y 
  MAP  y 


94/10/14
24
Example
 Probability density of observation given   
 e  y if y  0
p  y   
 0 if y  0
 The prior probability density of 
 e   if   0
w    
 0
 0 if   0
 the posterior probability density of  given Y=y
  e   y 
2
  y 



y

e


 
w  / y      e   y  d
0

0
if   0


where p  y   0  e
94/10/14
  y 
d     y 
if  and y  0
2
25
Example

The MMSE


0
0
ˆMMSE  y     w  / y  d      y   e
   y 
2


0
2e
  y 
2
  y 
d
d
 2   y 
o the Bayes risk is the average of the posterior cost

MMSE  r ˆMMSE

 
 
 E E    E  y Y 
 E E ˆMMSE Y    Y
2
2


 E Var   Y 
• the minimum MSE is the average of the conditional variance of
94/10/14

26
Example
• the conditional variance given Y=y is shown as
Var   y   E 2 Y  y   E 2  Y  y 
Var   y     2w  y  d  ˆMMSE  y 
0

   y 

2


0
 2   y 
•
2

 3w  y  d  4   y 
2

2

MMSE


MMSE  E Var   Y    Var   y  p  y dy

0

2
2
   2   y      y   dy
0 


 2 3 2
94/10/14
27
Example
 MMAE
o From the definition the MMAE estimate is the median of w  y 
o Because  is a continuous random variable given Y=y, the MMAE
estimate can be obtained

1
ˆABS  y  w  y d  2
2


ˆABS  y 
  y   e
   y 
d 
1
2
o By changing the variable x    y 



94/10/14
 y ˆABS  y 
xe  xdx 
1
2
  xe x  e  x 

  y ˆABS  y 

1
2

1
  y ˆABS  y 
ˆ

  y  ABS  y   1 e
2
28
Example
o the solution is
ˆABS  y  
T
y
• where T is the solution for 1  T  eT  1 2 ¸ and T~1.68.
 The MAP

max w  y =max   y   e y 

max  e
e 
   y 
  y 
2

   y  e

  y 

ˆMAP

0
1
y
o
94/10/14
29
Example
 multiple observation
p  y  
w  y  
n
e
 y k
 e
n
n
yk
k 1
k 1
w   p  y 
n 1 
   y 
e
p y 
n!
n

ˆMMSE    w  y  d 
0
ˆMAP 
94/10/14

n 1
n




y

k


k 1


n


   y k 


 k 1 


1
 0
1
n
n

yk  

 

  n k 1 n  
1
n

yk 


n

k 1 n 

30
Nonrandom parameter(real) estimation
 A problem in which we have a parameter indexing the class
of observation statistics, that is not modeled as a random
variable but nevertheless is unknown.
 Don’t have enough prior information about the parameter to
assign a prior probability distribution to it.
 Treat the estimation of such parameters in an organized
manner.
94/10/14
31
Statement of problem
 Given the observation Y=y, what is the best estimate of 
  is real and no information about the true value of 
 the only averaging of cost that we can be done is with
respect to the distribution of Y given  ,the conditional risk


R     E ˆ  y   
 

2
o we can not generally expect to minimize the conditional risk uniformly
o For any particular value of  , 0 the conditional mean-squared error
can be made zero by choosing ˆ  y  to be identically 0 for all
observations
o However, it can be poor if the value of 0 is not near the true value of 
94/10/14
32
Statement of problem



 E ˆ  y         
 E ˆ  y       E       2E ˆ  y        
E ˆ  y    0
2
2
0
2
2
0

0
 

o Unless   0 E ˆ  y   0 2  E ˆ  y    2
o it is not a good estimator due to not with minimum conditional meansquared-error。
 the conditional mean


E ˆ  y    ˆ  y  p  y  dy   
o if E ˆ  y    we say that the estimate is unbiased
o in general we have biased estimate b    E ˆ  y   
o variance of estimator var    E ˆ  y   E ˆ  y 2
o

94/10/14

b    0
33
MVUE

var    E ˆ  y   b    

2
   b    2E b   ˆ  y    

 E ˆ  y       b    2b   E ˆ  y    
 E ˆ  y       b  
 E ˆ  y   
2
2

2

bp   0

2
2
2
for the unbias estimator b    0 ,and the variance is the
conditional mean-squared error under p
var    E

94/10/14

ˆ  y   

2
The best we can hope for is minimum variance unbiased
estimator-MVUE
34
Q&A
94/10/14
35