Download September 22

ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Gaussian Likelihood
Recall the general mean-variance specification
E(Y |x) = f (x, β),
var(Y |x) = σ 2 g (β, θ, x)2 .
When g (·) is constant, assuming a Gaussian likelihood leads to OLS.
When g (·) is not constant, assuming a Gaussian likelihood with
known weights (i.e. g (·) depends only on x) leads to WLS.
When g (·) is not constant, and depends on unknown parameters,
assuming a Gaussian likelihood, what will happen?
1 / 19
Gaussian Likelihood and Quadratic Equations
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Estimating Equation for β
Log likelihood:
−n log σ −
n
X
j=1
n
1 X {Yj − f (xj , β)}2
.
log g (β, θ, xj ) −
2 j=1 σ 2 g (β, θ, xj )2
Differentiate w.r.t. β, equate to zero, and rearrange:
n
X
{Yj − f (xj , β)} fβ (xj , β)
g (β, θ, xj )2
j=1
"
#
n
2
X
{Y
−
f
(x
,
β)}
j
j
+ σ2
2 − 1 νβ (β, θ, xj ) = 0.
2
σ
g
(β,
θ,
x
)
j
j=1
2 / 19
Gaussian Likelihood and Quadratic Equations
Estimating Equation
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Here
νβ (β, θ, xj ) =
∂ log g (β, θ, xj )
gβ (β, θ, xj )
=
.
∂β
g (β, θ, xj )
The first term is the same as in GLS approach.
The second term arises from the dependence of var(Y |x) on β.
Note that we now need either to know σ 2 or estimate it in order to
estimate β (not separable).
3 / 19
Gaussian Likelihood and Quadratic Equations
Estimating Equation
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Comparison
The GLS equation:
n
X
{Yj − f (xj , β)} fβ (xj , β)
= 0.
σ 2 g (β, θ, xj )2
j=1
The normal theory ML equation:
n
X
{Yj − f (xj , β)} fβ (xj , β)
σ 2 g (β, θ, xj )2
j=1
"
#
n
X
{Yj − f (xj , β)}2
+
− 1 νβ (β, θ, xj ) = 0.
2 g (β, θ, x )2
σ
j
j=1
4 / 19
Gaussian Likelihood and Quadratic Equations
Estimating Equation
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Remarks
Both equations are unbiased. That is, the expectation of each
summand in each equation is 0, as both equations are evaluated at
the true values of β, θ and σ 2 .
The summand in the GLS equation is a linear function of Yj , while
the summand in the ML equation is a quadratic function of Yj .
If the data are really Gaussian, normal theory ML is (asymptotically)
efficient and GLS is inefficient (to be shown later).
If the data are not really Gaussian, either may be better than the
other.
5 / 19
Gaussian Likelihood and Quadratic Equations
Estimating Equation
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
General Estimating Equation Framework
Both GLS and normal theory ML equations may be written as
n
X
DTj Vj−1 (sj − mj ) = 0 .
j=1
(p×1)
Here
sj = “response”
mj = “mean function”
Dj = “gradient of mean function”
Vj−1 = “weights” = {var ( sj | xj )}−1 .
6 / 19
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
GLS:
sj = Yj
mj = f (xj , β)
Dj = fβ (xj , β)T
Vj−1 =
7 / 19
1
.
σ 2 g (β, θ, xj )2
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Normal theory ML:
sj =
Yj
{Yj − f (xj , β)}2
f (xj , β)
mj =
σ 2 g (β, θ, xj )2
fβ (xj , β)T
Dj =
2σ 2 g (β, θ, xj )2 νβ (β, θ, xj )T
2
−1
σ g (β, θ, xj )2
0
−1
Vj =
.
0
2σ 4 g (β, θ, xj )4
8 / 19
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Derivation of Vj
Two useful facts:
If ∼ N(0, 1), then
var
2
=
1 0
0 2
.
If has mean 0, variance 1, skewness 0, and (excess) kurtosis κ,
then
1
0
var
=
2
0 2+κ
Define
j =
9 / 19
Yj − f (xj , β)
σg (β, θ, xj )
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall:
Vj = var ( sj | xj )
and
sj =
Yj
{Yj − f (xj , β)}2
Then
var[{Yj − f (xj , β)}2 |xj ] = σ 4 g (β, θ, xj )4 var(2 )
and
cov[Yj , {Yj − f (xj , β)}2 |xj ] = σ 3 g (β, θ, xj )3 E {j (2j − 1)}
10 / 19
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
If E (3j ), the coefficient of skewness, is nonzero, then Vj is no longer
diagonal.
If var(2j ) − 2 = E (4j ) − 3 = κj (the coefficient of excess kurtosis) is
not zero, then the multiplier 2 in the (2, 2) element of Vj is replaced
by 2 + κ.
To maintain asymptotic efficiency (discussed later), the estimating
equations would need to be replaced by
n
X
{Yj − f (xj , β)} fβ (xj , β)
σ 2 g (β, θ, xj )2
j=1
"
#
n
2 X {Yj − f (xj , β)}2
+
− 1 νβ (β, θ, xj ) = 0.
2 + κ j=1
σ 2 g (β, θ, xj )2
11 / 19
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The most common departure from the normal distribution is heavy
tails (as in the t-distribution and other scale-mixtures of Gaussian
densities).
In this case, κ > 0, so the quadratic term in the estimating equation
is down-weighted.
But note: kurtosis is hard to estimate except in very large samples,
so we would usually not know how to weight the quadratic term.
12 / 19
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Example
For the (standardized) t-distribution with degrees of freedom ν > 4,
the excess kurtosis is
6
κ=
.
ν−4
If ν = 7, then κ = 2, and the quadratic term’s weight should be
halved.
Note
If T ∼ tν with ν > 2, then
r
T∗ =
13 / 19
ν−2
T ∼ tν,standardized .
ν
Gaussian Likelihood and Quadratic Equations
General Estimating Equation Framework
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Estimating σ 2
Gaussian MLE
The likelihood equation for σ 2 is
"
#
n
2
X
{Yj − f (xj , β)}
1
− 1 = 0.
σ j=1
σ 2 g (β, θ, xj )2
Equivalently,
"
#
n
o
X
{Yj − f (xj , β)}2 − σ 2 g (β, θ, xj )2 n
2
2σg
(β,
θ,
x
)
= 0.
j
4 g (β, θ, x )4
2σ
j
j=1
14 / 19
Gaussian Likelihood and Quadratic Equations
Estimating σ 2
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall: normal theory ML equations for β with known σ 2 :
n
X
DTj Vj−1 (sj − mj ) = 0
j=1
(p×1)
where
Yj
sj =
{Yj − f (xj , β)}2
f (xj , β)
mj =
σ 2 g (β, θ, xj )2
fβ (xj , β)T
Dj =
2σ 2 g (β, θ, xj )2 νβ (β, θ, xj )T
2
−1
σ g (β, θ, xj )2
0
−1
Vj =
.
0
2σ 4 g (β, θ, xj )4
15 / 19
Gaussian Likelihood and Quadratic Equations
Estimating σ 2
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
To estimate σ 2 , we have an additional equation with
sj = {Yj − f (xj , β)}2
mj = σ 2 g (β, θ, xj )2
Dj = 2σg (β, θ, xj )2
n
o−1
4
−1
4
Vj = 2σ g (β, θ, xj )
.
16 / 19
Gaussian Likelihood and Quadratic Equations
Estimating σ 2
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
We can incorporate the equation for σ 2 into those for β by:
adding a further column to Dj :
Dj =
fβ (xj , β)T
0
2
T
2
2σ g (β, θ, xj ) νβ (β, θ, xj ) 2σg (β, θ, xj )2
extending the right hand side to
.
0
((p+1)×1)
17 / 19
Gaussian Likelihood and Quadratic Equations
Estimating σ 2
.
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
GLS
The GLS equations may also be extended to include an equation
for σ 2 , if we use the biased ML-type estimator.
We use the same sj , mj , and Vj−1 as for the normal theory MLEs, but
Dj =
fβ (xj , β)T
0
0
2σg (β, θ, xj )2
.
Of course, because Dj is diagonal, the equations do separate, and σ 2
cancels out of the equations for β.
18 / 19
Gaussian Likelihood and Quadratic Equations
Estimating σ 2
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Comparison
Gaussian ML:
2 2
−1 n X
fβj 2σ 2 gj2 νβj
σ gj
0
Yj − fj
(Yj − fj )2 − σ 2 gj2
0
2σgj2
0
2σ 4 gj4
j=1
= 0.
GLS:
2 2
−1 n X
σ gj
0
Yj − fj
fβj
0
0 2σgj2
(Yj − fj )2 − σ 2 gj2
0
2σ 4 gj4
j=1
= 0.
19 / 19
Gaussian Likelihood and Quadratic Equations
Estimating σ 2