Download STAT 361 – Fall 2016

STAT 361 – Fall 2016
Homework 3: Solution
1. (Regression through the origin) Let data (x1 , y1 ), ..., (xn , yn ) satisfy the assumption
of the regression through the origin model:
yi = dxi + ei ,
i = 1, ..., n,
where d is unknown slope parameter, xi are given constants, and the measurement
errors are independent random variables, with Eei = 0 and Varei = σ 2 . Let dˆ be the
least squares estimate; cf. Homework 1, Problem 4b).
P
ˆ i )2 satisfies
a) Prove that the residual squared error SSres = ni=1 (yi − dx
SSres =
n
X
yi2
i=1
P
( ni=1 xi yi )2
− Pn 2 .
i=1 xi
b) Find the expected value ESSres .
c) Conclude that
s2 =
1
SSres
n−1
is an unbiased estimate of σ 2 .
Solution: a) The least squares estimate dˆ of the slope d was found to be
Pn
xi y i
dˆ = Pi=1
.
n
2
i=1 xi
Thus,
SSres =
n
X
yi2 −2dˆ
i=1
n
X
xi yi +dˆ2
i=1
n
X
i=1
yi2
n
X
x2i
=
i=1
n
X
yi2 −2
i=1
µPn
¶ n
µPn
¶2 n
xi y i X
xi y i X 2
i=1
i=1
Pn 2
xi yi + Pn 2
xi =
x
i
i=1 xi
i=1
i=1
i=1
P
P
P
n
( ni=1 xi yi )2 ( ni=1 xi yi )2 X 2 ( ni=1 xi yi )2
yi − Pn 2 .
− 2 Pn 2 + Pn 2 =
i=1 xi
i=1 xi
i=1 xi
i=1
b) Using properties of expectations and variances, we get
Pn
Pn
P
Pn
xi E (dxi + ei )
xi · (dxi )
d ni=1 x2i
xi E y i
i=1
i=1
i=1
ˆ
Pn 2
Pn 2
=
= Pn 2 = d,
Ed = Pn 2 =
i=1 xi
i=1 xi
i=1 xi
i=1 xi
and
Var dˆ =
Pn
i=1
x2i Var yi
P
2
( ni=1 x2i )
1
Pn
=
i=1
x2i Var (dxi + ei )
Pn 2
=
i=1 xi
Pn 2
P
xi Var ei
σ 2 ni=1 x2i
σ2
i=1
Pn 2
= Pn 2 = Pn 2 .
i=1 xi
i=1 xi
i=1 xi
Moreover,
Eyi2 = Varyi + (Eyi )2 = σ 2 + d2 x2i .
Thus by Part a),
ESSres =
n
X
µ
(σ
2
i=1
+d2 x2i )−
σ2
Pn
2
i=1 xi
2
+d
c) By Part b),
Es2 =
¶X
n
x2i
2
2
= (nσ +d
i=1
n
X
x2i )−(σ 2 +d2
i=1
n
X
x2i ) = (n−1)σ 2 .
i=1
1
ESSres = σ 2 ,
n−1
i.e., s2 is unbiased variance estimate.
2. (MLE) Let in the model discussed in Problem 1 the error terms ei have normal
distribution.
a) Explain why in this case the least squares estimate dˆ coincides with the maximum
likelihood estimate of the slope.
b) Assuming that σ is known, give a formula for the 95% confidence interval for the
ˆ
slope d, based on its estimate d.
Solution: a) Since under the assumption of normality, the responses yi are independent normal random variables N (dxi , σ 2 ), the joint pdf, or the likelihood function, of
y1 , ..., yn is
Ã
!
n
n
Y
y −dx )2
n
1
1 X
− i 2i
2
2
−
2
√
f (y1 , ..., yn |d, σ ) =
e 2σ
= (2πσ ) 2 exp − 2
(yi − dxi ) .
2
2σ i=1
2πσ
i=1
To find the MLE of the slope one has to maximize w.r.t. d the likelihood function or,
what is equivalent, the log-likelihood function
2
2 −n
2
2
l(d, σ ) = ln f (y1 , ..., yn |d, σ ) = ln(2πσ )
n
1 X
− 2
(yi − dxi )2 .
2σ i=1
P
This is equivalent to minimizing ni=1 (yi −dxi )2 . Thus, under the normality assumption
the LSE dˆ is at the same time the MLE.
b) Under the normality assumption, the LSE dˆ is a linear function of independent
normal random variables yi . Hence, dˆ itslef has a normal distribution. Moreover, by
Problem 1,
¶
µ
σ2
ˆ
d ∼ N d, Pn 2 .
i=1 xi
2
Through standardization this implies that
dˆ − d
q 2
Pnσ
i=1
∼
N (0, 1).
x2i
Thus, by definition of critical value z(α/2),

dˆ − d
P −z(α/2) ≤ q 2
Pnσ
i=1

≤ z(α/2) = 1 − α.
x2i
Since z(0.05/2) = 1.96, it follows that
s
s
Ã
!
2
2
σ
σ
P dˆ − 1.96 Pn 2 ≤ d ≤ dˆ + 1.96 Pn 2 = 0.95.
i=1 xi
i=1 xi
Thus, the required confidence interval is
s
dˆ ± 1.96
σ2
Pn
i=1
x2i
.
3. (Simple linear regression, continued) Consider the simple linear regression:
yi = a + bxi + ei ,
i = 1, ..., n.
Assume that the errors ei are independent random variables, with Eei = 0 and
Var ei = σ 2 . Let b̂ and â be the least squares estimates of the slope and the intercept,
respectively; cf. Homework 1, Problem 4c).
a) Show that b̂ in an unbiased estimate of the slope b.
Hint: Represent b̂ as a linear function of responses: b̂ =
Pn
i=1 ci yi .
b) Show that â is an unbiased estimate of the intercept a.
Hint: Use Part a) and properties of expectation.
c) Find the variance Var b̂.
Hint: Part a) and properties of the variance.
d) Find the variance Var â.
Pn
Hint: Represent â as a linear function of the random responses:
â
=
i=1 di yi , and
Pn
use the basic properties of variances. Recall the identity i=1 (xi − x̄) = 0.
e) Find the covariance Cov(â, b̂).
Hint: Recall the following property of covariances:
P
P
P
Cov ( ni=1 ci yi , ni=1 di yi ) = ni=1 ci di · Var yi .
3
f ) Assuming that ei have normal distribution, what is the distribution of the estimates
â and b̂?
Solution: a) The estimate b̂ of the slope b can be represented in the following form:
n
n
1 X
1 X
b̂ =
(xi − x̄)(yi − ȳ) =
(xi − x̄)yi .
Sxx i=1
Sxx i=1
(1)
Using the properties of expected values we find
Eb̂ =
n
n
n
1 X
1 X
b X
(xi − x̄)E yi =
(xi − x̄)(a + bxi ) =
(xi − x̄)xi =
Sxx i=1
Sxx i=1
Sxx i=1
n
b X
Sxx
(xi − x̄)(xi − x̄) = b
= b.
Sxx i=1
Sxx
Thus, b̂ is an unbiased estimate of b.
b) Since â = ȳ − b̂x̄, we get
n
Eâ = Eȳ − x̄Eb̂ =
n
1X
1X
Eyi − bx̄ =
(a + bxi ) − bx̄ = (a + bx̄) − bx̄ = a.
n i=1
n i=1
Thus, â is an unbiased estimate of a.
c) By the above equation (1) and properties of variance,
n
σ2 X
σ 2 Sxx
σ2
Varb̂ = 2
(xi − x̄)2 =
=
.
2
Sxx i=1
Sxx
Sxx
d) We can represent the estimate â of the intercept a in the form
n
1X
â = ȳ − b̂x̄ =
yi − x̄
n i=1
n
1X
yi − x̄
n i=1
Pn
i=1 (xi
− x̄)yi
Sxx
Pn
=
i=1 (xi
n µ
X
1
i=1
4
− x̄)(yi − ȳ)
=
Sxx
x̄(xi − x̄)
−
n
Sxx
¶
yi .
Thus,
Var â =
n µ
X
1
x̄(xi − x̄)
−
n
Sxx
i=1
¶2
· Var yi =
n µ
X
1
x̄(xi − x̄)
−
n
Sxx
i=1
¶2
σ2 =
¶
µ
¶
n µ
X
1
x̄(xi − x̄) x̄2 (xi − x̄)2
1
x̄2
2
σ
−2
=σ
+
.
+
2
2
n
nS
S
n
S
xx
xx
xx
i=1
2
e) By the hint,
Cov(â, b̂) =
n µ
X
1
x̄(xi − x̄)
−
n
Sxx
i=1
σ
2
n µ
X
1
i=1
x̄(xi − x̄)
−
n
Sxx
¶
¶
xi − x̄
Var yi2 =
Sxx
n
xi − x̄
σ 2 x̄ X
σ 2 x̄
=− 2
(xi − x̄)2 = −
.
Sxx
Sxx i=1
Sxx
f ) Since both estimates â and b̂ are linear functions of independent normal random
variables yi , they themselves have normal distribution. The parameters of these distributions have been already found in Parts a)-d):
â ∼ N (a, Var â),
b̂ ∼ N (b, Var b̂).
4. (Linear model with 2 regression parameters) Consider a linear model with two
regression parameters:
yi = aui + bvi + ei ,
i = 1, ..., n,
where, as usual, yi are the responses, and ui and vi are given non-random stimuli
variables. Find the least squares estimates of the parameters a and b.
Solution: a) To find the least squares estimates of the parameters a and b, we minimize the sum of sqaures
n
X
(yi − aui − bvi )2 .
i=1
By differentiation with respect to a and b and canceling the common factor −2, we
arrive at two linear equations,
n
X
(yi − aui − bvi )ui = 0,
i=1
n
X
(yi − aui − bvi )vi = 0.
i=1
5
Moving the unknown parameters to one side, we have
a
n
X
u2i
+b
i=1
a
n
X
n
X
ui vi =
i=1
ui vi + b
i=1
ui y i ,
i=1
n
X
i=1
n
X
vi2
=
n
X
vi yi .
i=1
Solving these linear equations in the usual way, we get find the least squares estimates
P
P
P
P
( ni=1 vi2 ) ( ni=1 ui yi ) − ( ni=1 ui vi ) ( ni=1 vi yi )
,
â =
P
P
P
2
( ni=1 u2i ) ( ni=1 vi2 ) − ( ni=1 ui vi )
and
b̂ =
(
Pn
P
P
P
u2i ) ( ni=1 vi yi ) − ( ni=1 vi2 ) ( ni=1 ui yi )
.
P
P
P
2
( ni=1 u2i ) ( ni=1 vi2 ) − ( ni=1 ui vi )
i=1
These LS estimates are uniquely defined, provided that the common denominator in
both expressions is not equal 0.
5. (Determining cargo weight) Upon entering a cargo terminal, a loaded truck was
weighed, and the result was Y1 . Before shipping, the truck was unloaded, and both the
truck and the cargo were weighed separately. The respective results were Y2 and Y3 .
Assume that the readings of weights were prone to independent Gaussian errors with
mean zero and variance σ 2 .
a) Set up a linear model for estimating the weights of the truck (T ) and the cargo (C).
b) Find the corresponding least squares estimates T̂ and Ĉ.
c) Find the variances of the least squares estimates T̂ and Ĉ.
d) Assuming that σ is known, exhibit 99% confidence intervals for the true weights, T
and C.
Solution: a)
Y1 = T + C + e 1 ,
Y2 = T +
Y3 =
e2 ,
C + e3 .
b) The problem can be reduced to the previous one, with a = T, b = C, n = 3 and
   
   
1
1
u1
v1
 u2  =  1  and  v2  =  0  ,
u3
v3
0
1
for which the general formula for LS estimates has been already found. Alternatively,
these estimates can be found directly by minimizing the sum of squares:
SS(T, C) = (Y1 − T − C)2 + (Y2 − T )2 + (Y3 − C)2 .
6
To minimize, we differentiate this sum of squares wit respect to T and C and set these
derivatives to 0:
−2(Y1 − T − C) − 2(Y2 − T ) = 0
=⇒
2T + C = Y1 + Y2 ,
−2(Y1 − T − C) − (Y3 − C) = 0,
=⇒
T + 2C = Y1 + Y3 .
Solving these two equations, we find the LSE’s
Y1 + 2Y2 − Y3
,
3
By the rules of expected values,
T̂ =
ET̂ =
Ĉ =
Y1 + 2Y3 − Y2
.
3
EY1 + 2EY2 − EY3
(T + C) + 2T − C
=
= T,
3
3
and similarly
EY1 + 2EY3 − EY2
(T + C) + 2C − T
=
= C,
3
3
i.e., these estimates are both unbiased.
EĈ =
c) Since VarYi = Varei = σ 2 ,
i = 1, 2, 3, using the properties of variances we get
6
2
Var T̂ = Var Ĉ = σ 2 = σ 2 .
9
3
d) Since Y1 , Y2 , Y3 are independent normal random variables, we find that
µ
¶
µ
¶
2σ 2
2σ 2
T̂ ∼ N T,
and Ĉ ∼ N C,
.
3
3
This allows to construct the corresponding 99% confidence intervals in the usual way,
as
r
r
2
2
T̂ ± z(0.01/2)σ
and Ĉ ± z(0.01/2)σ
3
3
where z(0.01/2) can be found from the Tables of normal distribution or using R:
z(0.01/2) = 2.58.
6. (Variance of Z 2 ) Let Z be standard normal random variable. Calculate Var(Z 2 ).
Solution: We know already that VarZ = EX 2 = 1. Next, we calculate using partial
integration
Z ∞
Z ∞ ³ 2 ´0
2
z
1
1
4
4 − z2
dz = − √
EZ = √
z e
z 3 e− 2 dz =
2π −∞
2π −∞
Z
¯∞
∞
z2 ¯
z2
1
1
− √ z 3 e− 2 ¯
+√
3z 2 e− 2 dz = 0 + 3 = 3.
−∞
2π
2π −∞
2
3
2
Thus, VarZ = EZ − (EZ) = 3 − 1 = 2.
7