Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Signal-flow graph wikipedia , lookup
Cubic function wikipedia , lookup
Quartic function wikipedia , lookup
Elementary algebra wikipedia , lookup
Quadratic form wikipedia , lookup
System of linear equations wikipedia , lookup
System of polynomial equations wikipedia , lookup
History of algebra wikipedia , lookup
ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Gaussian Likelihood Recall the general mean-variance specification E(Y |x) = f (x, β), var(Y |x) = σ 2 g (β, θ, x)2 . When g (·) is constant, assuming a Gaussian likelihood leads to OLS. When g (·) is not constant, assuming a Gaussian likelihood with known weights (i.e. g (·) depends only on x) leads to WLS. When g (·) is not constant, and depends on unknown parameters, assuming a Gaussian likelihood, what will happen? 1 / 19 Gaussian Likelihood and Quadratic Equations ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Estimating Equation for β Log likelihood: −n log σ − n X j=1 n 1 X {Yj − f (xj , β)}2 . log g (β, θ, xj ) − 2 j=1 σ 2 g (β, θ, xj )2 Differentiate w.r.t. β, equate to zero, and rearrange: n X {Yj − f (xj , β)} fβ (xj , β) g (β, θ, xj )2 j=1 " # n 2 X {Y − f (x , β)} j j + σ2 2 − 1 νβ (β, θ, xj ) = 0. 2 σ g (β, θ, x ) j j=1 2 / 19 Gaussian Likelihood and Quadratic Equations Estimating Equation ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Here νβ (β, θ, xj ) = ∂ log g (β, θ, xj ) gβ (β, θ, xj ) = . ∂β g (β, θ, xj ) The first term is the same as in GLS approach. The second term arises from the dependence of var(Y |x) on β. Note that we now need either to know σ 2 or estimate it in order to estimate β (not separable). 3 / 19 Gaussian Likelihood and Quadratic Equations Estimating Equation ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Comparison The GLS equation: n X {Yj − f (xj , β)} fβ (xj , β) = 0. σ 2 g (β, θ, xj )2 j=1 The normal theory ML equation: n X {Yj − f (xj , β)} fβ (xj , β) σ 2 g (β, θ, xj )2 j=1 " # n X {Yj − f (xj , β)}2 + − 1 νβ (β, θ, xj ) = 0. 2 g (β, θ, x )2 σ j j=1 4 / 19 Gaussian Likelihood and Quadratic Equations Estimating Equation ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Remarks Both equations are unbiased. That is, the expectation of each summand in each equation is 0, as both equations are evaluated at the true values of β, θ and σ 2 . The summand in the GLS equation is a linear function of Yj , while the summand in the ML equation is a quadratic function of Yj . If the data are really Gaussian, normal theory ML is (asymptotically) efficient and GLS is inefficient (to be shown later). If the data are not really Gaussian, either may be better than the other. 5 / 19 Gaussian Likelihood and Quadratic Equations Estimating Equation ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response General Estimating Equation Framework Both GLS and normal theory ML equations may be written as n X DTj Vj−1 (sj − mj ) = 0 . j=1 (p×1) Here sj = “response” mj = “mean function” Dj = “gradient of mean function” Vj−1 = “weights” = {var ( sj | xj )}−1 . 6 / 19 Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response GLS: sj = Yj mj = f (xj , β) Dj = fβ (xj , β)T Vj−1 = 7 / 19 1 . σ 2 g (β, θ, xj )2 Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Normal theory ML: sj = Yj {Yj − f (xj , β)}2 f (xj , β) mj = σ 2 g (β, θ, xj )2 fβ (xj , β)T Dj = 2σ 2 g (β, θ, xj )2 νβ (β, θ, xj )T 2 −1 σ g (β, θ, xj )2 0 −1 Vj = . 0 2σ 4 g (β, θ, xj )4 8 / 19 Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Derivation of Vj Two useful facts: If ∼ N(0, 1), then var 2 = 1 0 0 2 . If has mean 0, variance 1, skewness 0, and (excess) kurtosis κ, then 1 0 var = 2 0 2+κ Define j = 9 / 19 Yj − f (xj , β) σg (β, θ, xj ) Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Recall: Vj = var ( sj | xj ) and sj = Yj {Yj − f (xj , β)}2 Then var[{Yj − f (xj , β)}2 |xj ] = σ 4 g (β, θ, xj )4 var(2 ) and cov[Yj , {Yj − f (xj , β)}2 |xj ] = σ 3 g (β, θ, xj )3 E {j (2j − 1)} 10 / 19 Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response If E (3j ), the coefficient of skewness, is nonzero, then Vj is no longer diagonal. If var(2j ) − 2 = E (4j ) − 3 = κj (the coefficient of excess kurtosis) is not zero, then the multiplier 2 in the (2, 2) element of Vj is replaced by 2 + κ. To maintain asymptotic efficiency (discussed later), the estimating equations would need to be replaced by n X {Yj − f (xj , β)} fβ (xj , β) σ 2 g (β, θ, xj )2 j=1 " # n 2 X {Yj − f (xj , β)}2 + − 1 νβ (β, θ, xj ) = 0. 2 + κ j=1 σ 2 g (β, θ, xj )2 11 / 19 Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response The most common departure from the normal distribution is heavy tails (as in the t-distribution and other scale-mixtures of Gaussian densities). In this case, κ > 0, so the quadratic term in the estimating equation is down-weighted. But note: kurtosis is hard to estimate except in very large samples, so we would usually not know how to weight the quadratic term. 12 / 19 Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Example For the (standardized) t-distribution with degrees of freedom ν > 4, the excess kurtosis is 6 κ= . ν−4 If ν = 7, then κ = 2, and the quadratic term’s weight should be halved. Note If T ∼ tν with ν > 2, then r T∗ = 13 / 19 ν−2 T ∼ tν,standardized . ν Gaussian Likelihood and Quadratic Equations General Estimating Equation Framework ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Estimating σ 2 Gaussian MLE The likelihood equation for σ 2 is " # n 2 X {Yj − f (xj , β)} 1 − 1 = 0. σ j=1 σ 2 g (β, θ, xj )2 Equivalently, " # n o X {Yj − f (xj , β)}2 − σ 2 g (β, θ, xj )2 n 2 2σg (β, θ, x ) = 0. j 4 g (β, θ, x )4 2σ j j=1 14 / 19 Gaussian Likelihood and Quadratic Equations Estimating σ 2 ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Recall: normal theory ML equations for β with known σ 2 : n X DTj Vj−1 (sj − mj ) = 0 j=1 (p×1) where Yj sj = {Yj − f (xj , β)}2 f (xj , β) mj = σ 2 g (β, θ, xj )2 fβ (xj , β)T Dj = 2σ 2 g (β, θ, xj )2 νβ (β, θ, xj )T 2 −1 σ g (β, θ, xj )2 0 −1 Vj = . 0 2σ 4 g (β, θ, xj )4 15 / 19 Gaussian Likelihood and Quadratic Equations Estimating σ 2 ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response To estimate σ 2 , we have an additional equation with sj = {Yj − f (xj , β)}2 mj = σ 2 g (β, θ, xj )2 Dj = 2σg (β, θ, xj )2 n o−1 4 −1 4 Vj = 2σ g (β, θ, xj ) . 16 / 19 Gaussian Likelihood and Quadratic Equations Estimating σ 2 ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response We can incorporate the equation for σ 2 into those for β by: adding a further column to Dj : Dj = fβ (xj , β)T 0 2 T 2 2σ g (β, θ, xj ) νβ (β, θ, xj ) 2σg (β, θ, xj )2 extending the right hand side to . 0 ((p+1)×1) 17 / 19 Gaussian Likelihood and Quadratic Equations Estimating σ 2 . ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response GLS The GLS equations may also be extended to include an equation for σ 2 , if we use the biased ML-type estimator. We use the same sj , mj , and Vj−1 as for the normal theory MLEs, but Dj = fβ (xj , β)T 0 0 2σg (β, θ, xj )2 . Of course, because Dj is diagonal, the equations do separate, and σ 2 cancels out of the equations for β. 18 / 19 Gaussian Likelihood and Quadratic Equations Estimating σ 2 ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Comparison Gaussian ML: 2 2 −1 n X fβj 2σ 2 gj2 νβj σ gj 0 Yj − fj (Yj − fj )2 − σ 2 gj2 0 2σgj2 0 2σ 4 gj4 j=1 = 0. GLS: 2 2 −1 n X σ gj 0 Yj − fj fβj 0 0 2σgj2 (Yj − fj )2 − σ 2 gj2 0 2σ 4 gj4 j=1 = 0. 19 / 19 Gaussian Likelihood and Quadratic Equations Estimating σ 2