Download 6. Finite Sample Properties of the Least Squares Estimator

Econometrics I
Professor William Greene
Stern School of Business
Department of Economics
6-1/49
Part 6: Estimating the Variance of b
Econometrics I
Part 6 – Estimating
the Variance of b
6-2/49
Part 6: Estimating the Variance of b
6-3/49
Part 6: Estimating the Variance of b
Econometric Dire Emergency
6-4/49
Part 6: Estimating the Variance of b
Context
The true variance of b|X is 2(XX)-1 . We
consider how to use the sample data to estimate
this matrix. The ultimate objectives are to form
interval estimates for regression slopes and to
test hypotheses about them. Both require
estimates of the variability of the distribution.
6-5/49
Part 6: Estimating the Variance of b
Estimating 2
Using the residuals instead of the disturbances:
The natural estimator: ee/N as a sample
surrogate for /n
Imperfect observation of i, ei = i - ( - b)xi
Downward bias of ee/N. We obtain the result
E[ee|X] = (N-K)2
6-6/49
Part 6: Estimating the Variance of b
Expectation of ee
e
 y - Xb
 y  X ( X ' X) 1 X ' y
1
 [I  X( X ' X) X ']y
 My  M( X  )  MX  M  M
e'e  (M'(M
 'M'M  'MM  'M
6-7/49
Part 6: Estimating the Variance of b
Method 1:
E[e'e|X ]  E'M |X 
E[ trace ('M |X ) ] scalar = its trace
E[ trace (M'|X ) ] permute in trace
[ trace E (M'|X ) ] linear operators
[ trace M E ('|X ) ] conditioned on X
[ trace M 2IN ] model assumption
2 [trace M ] scalar multiplication and I matrix
2 trace [IN - X ( X'X )-1 X' ]
2 {trace [IN ] - trace[X( X'X)-1 X' ]}
2 {N - trace[( X'X )-1 X'X ]} permute in trace
2 {N - trace[IK ]}
2 {N - K}
Notice that E[ee|X] is not a function of X.
6-8/49
Part 6: Estimating the Variance of b
Estimating σ2
The unbiased estimator is s2 = ee/(N-K).
“Degrees of freedom correction”
Therefore, the unbiased estimator of 2 is
s2 = ee/(N-K)
6-9/49
Part 6: Estimating the Variance of b
Method 2: Some Matrix Algebra
E[e'e|X ]  2 trace M
What is the trace of M? M is idempotent, so its
trace equals its rank. Its rank equals the number
of nonzero characeristic roots.
Characteric Roots :
Signature of a Matrix = Spectral Decomposition
= Eigen (own) value Decomposition
A = CC' where
C = a matrix of columns such that CC' = C'C = I
 = a diagonal matrix of the characteristic roots
elements of  may be zero
6-10/49
Part 6: Estimating the Variance of b
Decomposing M
Useful Result: If A = CC' is the spectral
decomposition, then A 2  C2 C ' (just multiply)
M = M2 , so  2  . All of the characteristic
roots of M are 1 or 0. How many of each?
trace(A ) = trace(CC')=trace(C'C)=trace( )
Trace of a matrix equals the sum of its characteristic
roots. Since the roots of M are all 1 or 0, its trace is
just the number of ones, which is N-K as we saw.
6-11/49
Part 6: Estimating the Variance of b
Example: Characteristic Roots of a
Correlation Matrix
6-12/49
Part 6: Estimating the Variance of b
R = CΛC  i 1 icici
6
6-13/49
Part 6: Estimating the Variance of b
Gasoline Data
6-14/49
Part 6: Estimating the Variance of b
X’X and its Roots
6-15/49
Part 6: Estimating the Variance of b
Var[b|X]
Estimating the Covariance Matrix for b|X
The true covariance matrix is 2 (X’X)-1
The natural estimator is s2(X’X)-1
“Standard errors” of the individual coefficients are
the square roots of the diagonal elements.
6-16/49
Part 6: Estimating the Variance of b
X’X
(X’X)-1
s2(X’X)-1
6-17/49
Part 6: Estimating the Variance of b
Standard Regression Results
---------------------------------------------------------------------Ordinary
least squares regression ........
LHS=G
Mean
= 226.09444
Standard deviation
=
50.59182
Number of observs.
=
36
Model size
Parameters
=
7
Degrees of freedom
=
29
Residuals
Sum of squares
= 778.70227
Standard error of e =
5.18187 <= sqr[778.70227/(36 – 7)]
Fit
R-squared
=
.99131
Adjusted R-squared
=
.98951
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-7.73975
49.95915
-.155
.8780
PG|
-15.3008***
2.42171
-6.318
.0000
2.31661
Y|
.02365***
.00779
3.037
.0050
9232.86
TREND|
4.14359**
1.91513
2.164
.0389
17.5000
PNC|
15.4387
15.21899
1.014
.3188
1.67078
PUC|
-5.63438
5.02666
-1.121
.2715
2.34364
PPT|
-12.4378**
5.20697
-2.389
.0236
2.74486
--------+-------------------------------------------------------------
6-18/49
Part 6: Estimating the Variance of b
The Variance of OLS - Sandwiches
If Var[] = 2 I, then Var[b|X] = 2 (X'X) -1
What if Var[]  2 I?
Possibilities: Heteroscedasticity, Autocorrelation, Clustering and common effects.
b =  + (X'X) -1 *  i 1 xi i
n
n
Var[b|X] = (X'X) -1  Var   i 1 xi i   (X'X)-1


= A sandwich matrix. = A B A
What does the variance of the sum (the meat) look like?
Leading cases.
(1) Heteroscedasticity
(2) Autocorrelation
(3) Grouped (clustered) observations with common effects.
6-19/49
Part 6: Estimating the Variance of b
Robust Covariance Estimation
Not a structural estimator of XX/n
 If the condition is present, the estimator
estimates the true variance of the OLS estimator
 If the condition is not present, the estimator
estimates the same matrix that (2/n)(X’X/n)-1
estimates .




6-20/49
Heteroscedasticity
Autocorrelation
Common effects
Part 6: Estimating the Variance of b
Heteroscedasticity Robust Covariance Matrix




Robust estimation: Generality
How to estimate Var[b|X] = 2 (X’X)-1 XX (X’X)-1 for
the LS b?
The distinction between estimating
2 an n by n matrix
and estimating the KxK matrix
2 XX = 2 ijij xi xj
NOTE…… VVVIRs for modern applied econometrics.


6-21/49
The White estimator
Newey-West.
Part 6: Estimating the Variance of b
6-22/49
Part 6: Estimating the Variance of b
The White Estimator
n
Est.Var[b]  ( X'X) 1   i1 ei2 x i x i' ( X'X) 1


2
e
 i1 i
n
Use
2

ˆ 
n
nei2 ˆ
ˆ )=n

ˆ i = 2 , Ω=diag(
ˆ i ) note tr(Ω

ˆ
1
2
ˆ   X'X 

ˆ  X'X   X'ΩX
Est.Var[b] 

 


n  n   n   n 
 ˆ 
2 X'ΩX
2  X'ΩX 
Does



ˆ 
 n   0?
 n 




6-23/49
Part 6: Estimating the Variance of b
Groupwise Heteroscedasticity
Countries
are ordered
by the
standard
deviation of
their 19
residuals.
Regression of log of per capita gasoline use on log of per capita income,
gasoline price and number of cars per capita for 18 OECD countries for 19
years. The standard deviation varies by country. The “solution” is
“weighted least squares.”
6-24/49
Part 6: Estimating the Variance of b
White Estimator
+--------+--------------+----------------+--------+--------+----------+
|Variable| Coefficient | Standard Error |t-ratio |P[|T|>t]| Mean of X|
+--------+--------------+----------------+--------+--------+----------+
Constant|
2.39132562
.11693429
20.450
.0000
LINCOMEP|
.88996166
.03580581
24.855
.0000 -6.13942544
LRPMG
|
-.89179791
.03031474
-29.418
.0000
-.52310321
LCARPCAP|
-.76337275
.01860830
-41.023
.0000 -9.04180473
| White heteroscedasticity robust covariance matrix |
+----------------------------------------------------+
Constant|
2.39132562
.11794828
20.274
.0000
LINCOMEP|
.88996166
.04429158
20.093
.0000 -6.13942544
LRPMG
|
-.89179791
.03890922
-22.920
.0000
-.52310321
LCARPCAP|
-.76337275
.02152888
-35.458
.0000 -9.04180473
6-25/49
Part 6: Estimating the Variance of b
Autocorrelated Residuals
logG=β1 + β2logPg + β3logY + β4logPnc + β5logPuc + ε
6-26/49
Part 6: Estimating the Variance of b
The Newey-West Estimator
Robust to Autocorrelation
Heteroscedasticity Component - Diagonal Elements
1 n 2
S 0   i1 ei x i x i'
n
Autocorrelation Component - Off Diagonal Elements
1 L
n
w le t e t l ( x t x t l  x t l x t )

l1  t l1
n
l
wl  1 
= "Bartlett weight"
L 1
S1 
1
1  X'X 
 X'X 
Est.Var[b]= 
[
S

S
]
0
1 

n  n 
 n 
6-27/49
1
Part 6: Estimating the Variance of b
Newey-West Estimate
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-21.2111***
.75322
-28.160
.0000
LP|
-.02121
.04377
-.485
.6303
3.72930
LY|
1.09587***
.07771
14.102
.0000
9.67215
LPNC|
-.37361**
.15707
-2.379
.0215
4.38037
LPUC|
.02003
.10330
.194
.8471
4.10545
--------+--------------------------------------------------------------------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
Robust VC
Newey-West, Periods =
10
--------+------------------------------------------------------------Constant|
-21.2111***
1.33095
-15.937
.0000
LP|
-.02121
.06119
-.347
.7305
3.72930
LY|
1.09587***
.14234
7.699
.0000
9.67215
LPNC|
-.37361**
.16615
-2.249
.0293
4.38037
LPUC|
.02003
.14176
.141
.8882
4.10545
--------+-------------------------------------------------------------
6-28/49
Part 6: Estimating the Variance of b
Panel Data

Presence of omitted effects
y it =x itβ + c i + εit , observation for person i at time t
y i = X iβ + c ii + ε i , Ti observations in group i
=X iβ + c i + ε i , note c i  (c i , c i ,...,c i )
y =Xβ + c + ε , Ni=1 Ti observations in the sample
Potential bias/inconsistency of OLS – depends
on the assumptions about unobserved c.
 Variance of OLS is affected by autocorrelation in
most cases.

6-29/49
Part 6: Estimating the Variance of b
Estimating the Sampling Variance of b

s2(X ́X)-1? Inappropriate because



A ‘robust’ covariance matrix



6-30/49
Correlation across observations (certainly)
Heteroscedasticity (possibly)
Robust estimation (in general)
The White estimator
A Robust estimator for OLS.
Part 6: Estimating the Variance of b
Cluster Robust Estimator
Robust variance estimator for Var[b]
Est.Var[b]
= ( X'X ) 1 Ni=1 ( X iei )(ei X i )  ( X'X ) 1
i
i
x it eit )  ( X'X) 1
x it eit )(Ct=1
= ( X'X ) 1 Ni=1 (Ct=1


i
i
eit eis x it x is  ( X'X ) 1
Cs=1
= ( X'X ) 1 Ni=1 Ct=1


e  a least squares residual
(If Ci = 1, this is the White estimator.)
6-31/49
Part 6: Estimating the Variance of b
6-32/49
Part 6: Estimating the Variance of b
6-33/49
Part 6: Estimating the Variance of b
Alternative OLS Variance Estimators
Cluster correction increases SEs
+---------+--------------+----------------+--------+---------+
|Variable | Coefficient | Standard Error |b/St.Er.|P[|Z|>z] |
+---------+--------------+----------------+--------+---------+
Constant
5.40159723
.04838934
111.628
.0000
EXP
.04084968
.00218534
18.693
.0000
EXPSQ
-.00068788
.480428D-04
-14.318
.0000
OCC
-.13830480
.01480107
-9.344
.0000
SMSA
.14856267
.01206772
12.311
.0000
MS
.06798358
.02074599
3.277
.0010
FEM
-.40020215
.02526118
-15.843
.0000
UNION
.09409925
.01253203
7.509
.0000
ED
.05812166
.00260039
22.351
.0000
Robust
Constant
5.40159723
.10156038
53.186
.0000
EXP
.04084968
.00432272
9.450
.0000
EXPSQ
-.00068788
.983981D-04
-6.991
.0000
OCC
-.13830480
.02772631
-4.988
.0000
SMSA
.14856267
.02423668
6.130
.0000
MS
.06798358
.04382220
1.551
.1208
FEM
-.40020215
.04961926
-8.065
.0000
UNION
.09409925
.02422669
3.884
.0001
ED
.05812166
.00555697
10.459
.0000
6-34/49
Part 6: Estimating the Variance of b
Bootstrapping
Some assumptions that underlie it - the sampling mechanism
Method:
1. Estimate using full sample: --> b
2. Repeat R times:
Draw N observations from the n, with replacement
Estimate  with b(r).
3. Estimate variance with
V = (1/R)r [b(r) - b][b(r) - b]’
6-35/49
Part 6: Estimating the Variance of b
Bootstrap Application
matr;bboot=init(3,21,0.)$
name;x=one,y,pg$
regr;lhs=g;rhs=x$
calc;i=0$
Proc
regr;lhs=g;rhs=x;quietly$
matr;{i=i+1};bboot(*,i)=b$...
Endproc
exec;n=20;bootstrap=b$
matr;list;bboot' $
6-36/49
Store results here
Define X
Compute b
Counter
Define procedure
… Regression
Store b(r)
Ends procedure
20 bootstrap reps
Display results
Part 6: Estimating the Variance of b
Results of Bootstrap Procedure
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-79.7535***
8.67255
-9.196
.0000
Y|
.03692***
.00132
28.022
.0000
9232.86
PG|
-15.1224***
1.88034
-8.042
.0000
2.31661
--------+------------------------------------------------------------Completed
20 bootstrap iterations.
---------------------------------------------------------------------Results of bootstrap estimation of model.
Model has been reestimated
20 times.
Means shown below are the means of the
bootstrap estimates. Coefficients shown
below are the original estimates based
on the full sample.
bootstrap samples have
36 observations.
--------+------------------------------------------------------------Variable| Coefficient
Standard Error b/St.Er. P[|Z|>z]
Mean of X
--------+------------------------------------------------------------B001|
-79.7535***
8.35512
-9.545
.0000
-79.5329
B002|
.03692***
.00133
27.773
.0000
.03682
B003|
-15.1224***
2.03503
-7.431
.0000
-14.7654
--------+-------------------------------------------------------------
6-37/49
Part 6: Estimating the Variance of b
Bootstrap Replications
Full sample result
Bootstrapped sample
results
6-38/49
Part 6: Estimating the Variance of b
Results of C&R Bootstrap Estimation
6-39/49
Part 6: Estimating the Variance of b
Bootstrap variance for a
panel data estimator
 Panel Bootstrap =
Block Bootstrap
 Data set is N groups of
size Ti
 Bootstrap sample is N
groups of size Ti drawn
with replacement.
6-40/49
Part 6: Estimating the Variance of b
6-41/49
Part 6: Estimating the Variance of b
Quantile Regression:
Application of Bootstrap
Estimation
6-42/49
Part 6: Estimating the Variance of b
OLS vs. Least Absolute Deviations
---------------------------------------------------------------------Least absolute deviations estimator...............
Residuals
Sum of squares
=
1537.58603
Standard error of e =
6.82594
Fit
R-squared
=
.98284
--------+------------------------------------------------------------Variable| Coefficient
Standard Error b/St.Er. P[|Z|>z]
Mean of X
--------+------------------------------------------------------------|Covariance matrix based on
50 replications.
Constant|
-84.0258***
16.08614
-5.223
.0000
Y|
.03784***
.00271
13.952
.0000
9232.86
PG|
-17.0990***
4.37160
-3.911
.0001
2.31661
--------+------------------------------------------------------------Ordinary
least squares regression ............
Residuals
Sum of squares
=
1472.79834
Standard error of e =
6.68059
Standard errors are based on
Fit
R-squared
=
.98356
50 bootstrap replications
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-79.7535***
8.67255
-9.196
.0000
Y|
.03692***
.00132
28.022
.0000
9232.86
PG|
-15.1224***
1.88034
-8.042
.0000
2.31661
--------+-------------------------------------------------------------
6-43/49
Part 6: Estimating the Variance of b
Quantile Regression





Q(y|x,) = x,  = quantile
Estimated by linear programming
Q(y|x,.50) = x, .50  median regression
Median regression estimated by LAD (estimates same
parameters as mean regression if symmetric conditional
distribution)
Why use quantile (median) regression?



6-44/49
Semiparametric
Robust to some extensions (heteroscedasticity?)
Complete characterization of conditional distribution
Part 6: Estimating the Variance of b
Estimated Variance for
Quantile Regression
6-45/49

Asymptotic Theory

Bootstrap – an ideal application
Part 6: Estimating the Variance of b
Asymptotic Theory Based Estimator of Variance of Q - REG
Model : yi  βx i  ui , Q( yi | xi , )  βx i , Q[ui | xi , ]  0
Residuals: uˆ i  yi - βˆ x i
1
A 1 C A 1 

N
1
11
N
A = E[f u (0)xx] Estimated by  i 1
1| uˆi | B xi xi
N
B2
Bandwidth B can be Silverman's Rule of Thumb:
Asymptotic Variance:
1.06
 Q (uˆi | .75)  Q(uˆi | .25) 
Min
su ,


.2
N
1.349


(1- )
C = (1- ) E[ xx] Estimated by
XX
N
For  =.5 and normally distributed u, this all simplifies to
 2
1
su  XX  .
2
But, this is an ideal application for bootstrapping.
6-46/49
Part 6: Estimating the Variance of b
 = .25
 = .50
 = .75
6-47/49
Part 6: Estimating the Variance of b
6-48/49
Part 6: Estimating the Variance of b
6-49/49
Part 6: Estimating the Variance of b