Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Statistics 550 Notes 14 Reading: Sections 3.2 I. Computation of Bayes procedures for complex problems For nonconjugate priors, the posterior mean (which is the Bayes estimator under squared error loss) is not typically available in closed form. 2 2 Example 1: X 1 , , X n iid N ( , ) , known, and our prior on is a logistic (a, b) distribution: 1 exp{( a) / b} ( ) b [1 exp{( a) / b}]2 The logistic distribution is a more heavily tailed distribution than the normal distribution. Since X is sufficient, we can just compute the posterior given X . The posterior pdf is 1 ( X ) 2 1 e ( a ) / b exp 2 ( a ) / b 2 ] 2 / n 2 / n b [1 e 2 ( a ) / b 1 ( X ) 1 e 1 exp d 2 ( a ) / b 2 ] 2 / n 2 / n b [1 e 1 p( | X ) p( X | ) ( ) p( X ) The numerator is not proportional to any commonly used density function and the denominator is not evaluatable in closed form. 1 Monte Carlo methods can be used to sample from the posterior distribution to approximate the Bayes estimator. For discussion of Monte Carlo methods for Bayesian inference, see Bayesian Data Analysis by Gelman, Carlin, Stern and Rubin; these methods will be discussed in Stat 542 taught by Professor Jensen in the Spring. II. Improper Priors Bayes estimators are defined with respect to a proper prior distribution ( ) . Often a weighting function ( ) is considered that is not a probability distribution; this is called an improper prior. Example 2: X ~ N ( ,1) , ( ) 1 . Example 3: X ~ Binomial( p , n ) . Consider the prior ( p) p 1 (1 p)1 , 0 p 1 , which in some sense corresponds to a Beta(0,0) distribution. However, this is not a proper distribution because 1 0 p 1 (1 p)1 dp . For an improper prior, we can still consider the “Bayes” risk of a decision procedure: r ( ) R( , ) ( )d (1.1) 2 * An estimator ( x) is called a generalized Bayes estimator with respect to a weighting function ( ) (even if it is not a proper probability distribution) if it minimizes the “Bayes” risk (1.1) over all estimators. We can write the “Bayes” risk (1.1) as r ( ) l ( , ( X ) p( X | )dX ( )d l ( , ( X ) p( X | ) ( ) d dX A decision procedure ( X ) for which for all X , ( X ) arg min l ( , a) p( X | ) ( )d aA is a generalized Bayes estimator (this is the analogue of Proposition 3.2.1). Let s( | X ) p( X | ) ( ) p( X | ) ( )d (1.2) p( X | ) ( )d , then a arg min l ( , a) p( X | ) ( ) d if and only if a arg min l ( , a)s( | X )d . Note that if aA aA Sometimes s ( | X ) is a proper probability distribution even if ( ) is not a proper probability distribution, and then we can think of s ( | X ) as the “posterior” density 3 function of | X and l ( , a )s ( | X )d as the “posterior” risk so that an estimator which minimizes the “posterior” risk for each X is a generalized Bayes estimator. Example 3 continued: For X ~ Binomial( p, n) and the 1 1 prior ( p) p (1 p) , 0 p 1 , consider the generalized Bayes estimator under squared error loss. Error! Reference source not found. equals n X n X 1 1 p (1 p) p (1 p) X s( p | X ) for 0 p 1 1 n X n X 1 1 0 X p (1 p) p (1 p) dp For 1 X n 1, s ( p | X ) is a Beta( X , n X ) distribution, i.e., s ( p | X ) is a proper probability distribution, and the action of the generalized Bayes estimator with respect to squared error loss is the expected value of p under the X X distribution s ( p | X ) , which equals X n X n . For X 0 and X n , the “posterior” density s ( p | X ) is no longer proper but it can be shown that X arg min l ( , a) p( X | ) ( ) d . Thus, the n a A X p generalized Bayes estimator of is n for ( p) p 1 (1 p)1 . 4 Another useful variant of Bayes estimators are limits of Bayes estimators. A nonrandomized estimator ( x ) is a limit of Bayes estimators if there exists a sequence of proper priors Bayes estimators v with respect to these prior distributions such that v ( x ) ( x) for all x. v and Example 3 continued: For X ~ Binomial( p, n) , ( X ) X / n is a limit of Bayes estimators. Consider a Beta ( r , s ) prior (which is proper if r 0, s 0 ); the Bayes X r estimator is n r s . Consider the sequence of priors Beta (1,1), Beta (1/2,1/2),Beta(1/3,1/3),... Since X r X lim , we have that ( X ) X / n is a limit of r 0 n r s n s 0 Bayes estimators. From a Bayesian point of view, estimators that are limits of Bayes estimators are somewhat more desirable than generalized Bayes estimators (often estimators are both limit of Bayes estimators and generalized Bayes estimators as in Example 3). This is because, by construction, a limit of Bayes estimators must be close to a proper Bayes estimator. In contrast, a generalized Bayes estimator may not be close to any proper Bayes estimator. III. Admissibility of Bayes rules: 5 In general, Bayes rules are admissible. * Theorem : Suppose that is an interval and is a Bayes rule with respect to a prior density function ( ) such that ( ) 0 for all and R ( , d ) is a * continuous function of for all d . Then is admissible. Proof: The proof is by contradiction. Suppose that is inadmissible. There is then another estimate, , such that R( , * ) R( , ) for all and with strict inequality for some , say 0 . Since R( , *) R( , ) is a continuous function of , there is an 0 and an interval h such that R( , *) R( , ) for 0 h 0 h Then, * 0 h R( , *) R( , ) ( )d R( , *) R( , ) ( )d 0 h 0 h ( )d 0 0 h But this contradicts the fact that * is a Bayes rule because a Bayes rule has the property that B( *) B( ) R( , *) R( , ) ( )d 0 . The proof is complete. 6 The theorem can be regarded as both a positive and negative result. It is positive in that it identifies a certain class of estimates as being admissible, in particular, any Bayes estimate. It is negative in that there are apparently so many admissible estimates – one for every prior distribution that satisfies the hypotheses of the theorem – and some of these might make little sense. Complete class theorems characterize the class of all admissible estimators. Roughly the class of all admissible estimators for most models is the class of all Bayes and limit of Bayes estimators. 7