Download bootstrap 2019

The Bootstrap
Econometrics III Lecture Notes
Ke-Li Xu
Indiana University
September 13, 2019
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
1 / 47
Contents
Bootstrap bias correction
Bootstrap standard error
Bootstrap coe¢ cient-based test (and CI)
Bootstrap t-test (and percentile-t CI)
Example: linear regression model
I
I
I
I
Pairwise bootstrap and residual-based bootstrap
Restricted bootstrap
Bootstrapping F test
Parametric bootstrap
Permutation test
Bootstrap: Improve oneself by one’s own e¤orts
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
2 / 47
Bootstrap bias correction
Suppose the parameter of interest is θ.
The bias of an estimator b
θ is
τ = Eb
θ
θ.
E.g. θ can be a linear regression coe¢ cient, when the strict exogeneity
assumption is not satis…ed.
If τ is known, then we can construct an (infeasible) bias-corrected estimator
of θ :
bc ,inf
b
θ
= bθ τ.
bc ,inf
bc ,inf
b
θ
is unbiased: E b
θ
= θ.
We now want to estimate τ. The bootstrap provides a solution.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
3 / 47
Suppose the (raw) data are fZ1 , ..., Zn g, coming from the unknown
distribution F .
We can rewrite (making the dependence on F explicit)
τ = τ ( F ) = EF b
θ
θ (F ).
E.g. in the linear regression yi = xi0 θ + ui ,
θ = θ (F ) = (EF xi xi0 ) 1 EF xi yi .
The idea of bootstrap is to approximate F by Fb , the empirical CDF of the
data:
n
Fb (z ) = n 1 ∑ I (Zi
z ).
i =1
Intuitively, Fb assigns the probability mass 1/n to each data point Zi .
Thus τ is estimated by τ (Fb ).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
4 / 47
Denote the moments under Fb as E ( ), Var ( ), etc.
De…ne b
θ to be like b
θ, but in terms of data coming from Fb (instead of the
original data, which come from F ).
What is τ (Fb ) then?
τ (F )
τ (Fb )
= EF bθ
= E bθ
θ (F )
θ (Fb ) = E b
θ
b
θ.
Here we have used θ (Fb ) = b
θ, which is typically true in econometrics when b
θ
is an MM (method of moments) estimator.
E.g. in the linear regression,
θ (Fb )
b
θ
= ( E xi xi 0 )
1
n
E xi yi = ( ∑ xi xi0 ) 1
i =1
n
= ( ∑ xi xi 0 )
i =1
1
∑ xi yi ,
i =1
n
∑ xi yi ,
i =1
where random variables (yi , xi 0 )0 are from Fb .
A resampling approach is needed to obtain E b
θ .
Ke-Li Xu (Indiana University)
n
Bootstrap
September 13, 2019
5 / 47
θ by taking many random draws from the data.
Bootstrap: Approximate E b
I
I
I
Take n random draws (with replacement) from the raw data. These n draws
are called a bootstrap sample: fZ1 , ..., Zn g.
Repeat doing this B times. So we have B bootstrap samples:
fZ1 (b ), ..., Zn (b )g, b = 1, ..., B .
For each bootstrap sample, compute b
θ (b ), where b
θ (b ) is just like b
θ except
that the sample fZ1 (b ), ..., Zn (b )g, instead of the raw data, is used.
B should be large, so that the distribution of fb
θ (b ) : b = 1, ..., B g well
approximates the distribution of b
θ .
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
6 / 47
Then
E b
θ
1 B b
b
θ ,
θ (b )
B b∑
=1
So de…ne the estimator of τ as b
τ:
τ (Fb ) = E b
θ
b
θ
b
θ,b
τ
b
θ
The bias-corrected estimator of b
θ is formed as
bc
b
θ =b
θ
b
τ = 2b
θ
b
θ .
Lastly, on a theoretical note, the approximation of F using Fb is justi…ed by
Glivenko-Cantelli Theorem: if Zi is iid,
sup jFb (z )
z
Ke-Li Xu (Indiana University)
Bootstrap
p
F (z )j ! 0.
September 13, 2019
7 / 47
Bootstrap standard errors
The bootstrap method estimates the …nite-sample variance of b
θ (i.e. Var(b
θ))
by:
Var (b
θ ).
After re-sampling the data B times, Var (b
θ ) is appximated by
Vbθboot =
B
1
B
∑ (bθ
1 b =1
(b )
b
θ )(b
θ (b )
b
θ )0 .
Vbboot is called the bootstrap variance estimator.
θ
Bootstrap standard error (if θ is a scalar)
SEbθboot =
Ke-Li Xu (Indiana University)
h
B
b
B 1 ∑ b =1 ( θ (b )
1
Bootstrap
b
θ )2
i1 /2
.
September 13, 2019
8 / 47
Bootstrap test (coe¢ cient-based)
In the bias correction above, we use the re-sampling to approximate a
moment
b
E (b
θ
θ ).
In fact, the distribution of b
θ
Ke-Li Xu (Indiana University)
b
θ can be also useful.
Bootstrap
September 13, 2019
9 / 47
Consider a generic scalar parameter θ.
In the regression setting, θ can be βj (the j-th slope).
Consider H0 : θ = θ 0 .
For an estimator b
θ, based on the asymptotic normality result
d
0
1
/
2
n (b
θ θ ) ! N (0, v 2 ) (if the null is true), we have
n1 /2 (b
θ
θ0 )
Asy .
where vb is an estimator of v .
N (0, vb2 ),
(1)
The bootstrap provides an alternative way of approximating the
(…nite-sample) distribution of n1 /2 (b
θ θ 0 ), instead of using N (0, vb2 ), as in
(1).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
10 / 47
θ θ 0 ) is Gn (F ), where F is the true
Denote the true distribution of n1 /2 (b
CDF behind the data. Note that both b
θ and θ 0 (true value, if the null holds)
depend on F .
If we know Gn (F ), then we calculate its lower and upper quantiles, so that a
test can be formed.
But in general, Gn (F ) is unknown
Two methods to approximate Gn (F ):
I
I
the asymptotic method (using G∞ (F ))
the bootstrap method (using Gn (Fb )), where Fb is the empirical CDF of the
data. [This is commonly referred to as the nonparametric bootstrap.]
n 1/2 (b
θ
n
The distribution of n1 /2 (b
θ
Ke-Li Xu (Indiana University)
1/2
(bθ
θ0 )
exact
G n (F )
exact
b
θ)
Gn (Fb )
b
θ ) is obtained by re-sampling.
Bootstrap
September 13, 2019
11 / 47
Algorithm (a bootstrap test for θ = θ 0 )
Suppose the (raw) data are fZ1 , ..., Zn g.
As before, we generate B bootstrap samples: fZ1 (b ), ..., Zn (b )g,
b = 1, ..., B.
For each bootstrap sample, compute b
θ (b ).
N Let q (α) be the α th quantile of {n1 /2 (bθ (b )
b
θ ), b = 1, ..., B g. N
Two-sided test: Reject H0 at level 5% if
n1 /2 (b
θ
θ 0 ) > q (0.975)
or n1 /2 (b
θ
θ 0 ) < q (0.025).
(2)
One-sided test (e.g. H0 : θ = θ 0 vs. HA : θ > θ 0 ): Reject H0 at level 5% if
n1 /2 (b
θ
Ke-Li Xu (Indiana University)
θ 0 ) > q (0.95).
Bootstrap
September 13, 2019
12 / 47
d
We need θ 0 be the true value (under H0 ) for n1 /2 (b
θ θ 0 ) ! N (0, v 2 ) to
hold.
In the bootstrap world, b
θ is the true value. Thus θ 0 has no role in the
bootstrap statistic:
n1 /2 (b
θ (b ) b
θ ).
A common mistake is to use n1 /2 (b
θ
I
I
θ 0 ) as the bootstrap statistic.
If so, the critical value would also change with θ 0 (note that n 1/2 (b
θ θ0 )
changes with θ 0 ), thereby the test may have no power.
[we will revisit this point later when we show the bootstrap test is consistent
(i.e. the power goes to one under the alternative)]
This coe¢ cient-based bootstrap test (and the induced con…dence interval (3)
below) has the advantage of not having to estimate the standard error.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
13 / 47
The advantage of bootstrap is that we can easily take random draws from Fb
as many as we want. (We can only take one random draw from F , i.e. the
original data in hand).
The asymptotic validity is usually shown like this:
Gn (F )
Goal
&
G∞ (F )
Asym.
Gn (Fb )
Bootstrap
%
Thus, in order to approximate Gn (F ), the asymptotic method and bootstrap
are …rst-order equivalent.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
14 / 47
Note that for each method, we need another approximation.
I
I
For the asymptotic method, we estimate the asymptotic variance.
For bootstrap, we draw B samples (requiring B to be large).
Fb is referred to as the bootstrap world (F is the world we want to learn
about).
In the bootstrap world, b
θ (b ) b
θ is one draw.
The main advantage of the bootstrap procedure above is that we don’t need
to derive G∞ (F ). In many cases, we don’t need to estimate the asymptotic
variance v 2 .
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
15 / 47
Bootstrap con…dence interval (coe¢ cient-based)
The 95% CI is obtained by inverting the test (2) (collecting all values which
can not be rejected).
θ θ ) q (0.975)g, or
We then have fθ : q (0.025) n1 /2 (b
b
θ
n 1 /2 q (0.975)
θ
b
θ
n 1 /2 q (0.025).
(3)
The CI (3) has correct asymptotic coverage 95%.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
16 / 47
Percent CI
A commonly-used bootstrap CI is formed by simply using lower and upper
quantiles of fb
θ (b ) : b = 1, ..., B g.
This is called percentile CI.
The percentile CI can be writtten as
θ 2 [b
θ + n 1 /2 q (0.025), b
θ + n 1 /2 q (0.975)].
(4)
It is important to note that the CI (4) is in general di¤erent from (3). They
are the same only if the bootstrap distribution is symmetric around zero (so
that q (0.025) = q (0.975)).
Asymptotic justi…cation of the percentile CI needs G∞ (F ) to be symmetric.
Thus the CI (3) is a preferred choice.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
17 / 47
We now show that the percent CI has the claimed coverage asymptotically if
symmetry is assumed.
Suppose
n1 /2 (b
θ
n1 /2 (b
θ
d
θ ) ! ξ,
(5)
d
b
θ ) ! ξ.
(6)
Assume that the distribution of ξ is symmetric around zero. (e.g. consider a
regression model, then ξ is normal.)
In our previous notation, ξ is G∞ (F ).
Then
P
θ 2 Percentile CI
b
θ + n 1 /2 q (0.025)
=
P
=
!
P
q (0.975)
P ( qξ (0.975)
=
P (qξ (0.025)
symmetry
Ke-Li Xu (Indiana University)
θ
b
θ + n 1 /2 q (0.975)
n1 /2 (b
θ θ)
q (0.025)
ξ
qξ (0.025))
ξ
Bootstrap
qξ (0.975)) = 0.95.
September 13, 2019
18 / 47
Consistency of the bootstrap test
Under a particular model, showing both (5) and (6) (with the same ξ)
implies the bootstrap validity (asymptotically), i.e. the distribution of
b
n1 /2 (b
θ θ ) can be approximated by the distribution of n1 /2 (b
θ
θ ).
We will showcase this for di¤erent models later in the class (e.g. (7) below).
We can also show that the bootstrap test is consistent.
Suppose H0 : θ = θ 0 is wrong. That means θ true 6= θ 0 .
Note that (5) holds only for θ true (not θ 0 ), thus the test statistic diverges, i.e.
n1 /2 (b
θ
θ0 )
=
!
n1 /2 (b
θ
dξ
θ true ) + n1 /2 (θ true
+ n1 /2 (θ true
On the other hand, by (6), n1 /2 (b
θ
θ0 )
θ 0 ) ! ∞.
b
θ ) remains …xed (irrelavant to θ 0 ).
So the test will always reject, asymptotically.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
19 / 47
Bootstrap t-test
Consider H0 : θ = θ 0 .
The t-statistic is t (θ 0 ) = vb 1 (b
θ
θ 0 ).
The asymptotic method relies on
d
t (θ 0 ) ! N (0, 1), under H0 .
Let the distribution of t (θ 0 ) be Gn (F ). Then G∞ (F ) = N (0, 1).
The bootstrap approximates Gn (F ) by Gn (Fb ). The bootstrap t-stat:
t = vb
1
(bθ
b
θ ).
Let qt (α) be 100α% quantile of t (where the actual calculation needs B
bootstrap re-sampling).
The hypothesis H0 is rejected if t (θ 0 ) > qt (0.975) or t (θ 0 ) < qt (0.025).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
20 / 47
Percent-t CI
Percentile-t con…dence interval for θ:
[bθ
vb
θ
qt (0.975), b
vb
qt (0.025)].
It is obtained by inverting the bootstrap t-test:
(bθ θ 0 ) < qt (0.975)
vbqt (0.975) < θ 0 < b
θ vbqt (0.025)
qt (0.025)
b
θ
Ke-Li Xu (Indiana University)
< vb
Bootstrap
1
September 13, 2019
21 / 47
Bootstrapping Linear regression
Pairwise bootstrap
The model
yi = xi0 β + ui ,
where xi is k 1, and ui
(0, σ2 ).
β.
bi = yi xi0 b
The OLS: b
β = (∑ni=1 xi xi0 ) 1 (∑ni=1 xi yi ). OLS residuals u
0
0
Pairwise bootstrap: generate iid sample (yi , xi ) by taking random draws
from (yi , xi0 )0 .
Population regression slope in the bootstrap world is
( E xi xi 0 )
1
E xi yi = ( n 1
n
∑ xi xi0 )
i =1
1
(n
1
n
∑ xi yi ) = bβ
i =1
True residuals: ui = yi
xi 0 b
β, which equals u
bj (for some j)
Orthogonal condition: E xi ui = E xi (yi
xi 0 b
β) = 0.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
22 / 47
In the original data world,
n 1 /2 ( b
β
d
β) ! N (0, (Exi xi0 ) 1 (Exi xi0 ui2 )(Exi xi0 ) 1 ).
In the bootstrap world,
b
β
n
, ( ∑ xi xi 0 )
1
= ( ∑ xi xi 0 )
1
i =1
n
i =1
We will show that
n 1 /2 ( b
β
Ke-Li Xu (Indiana University)
n
( ∑ xi yi )
i =1
n
n
i =1
i =1
( ∑ xi ( xi 0 b
β + ui )) = b
β + ( ∑ xi xi 0 )
1
n
( ∑ xi u i )
i =1
d
b
β) ! N (0, (Exi xi0 ) 1 (Exi xi0 ui2 )(Exi xi0 ) 1 ).
Bootstrap
September 13, 2019
(7)
23 / 47
To show (7), note that
1 n
1 n
xi xi 0 = E xi xi 0 + op (1) = ∑ xi xi0 + op (1) = Exi xi0 + op (1)
∑
n i =1
n i =1
n 1 /2
n
∑ xi ui
i =1
d
d
! N (0, E xi xi 0 ui 2 ) ! N (0, Exi xi0 ui2 ),
(8)
by CLT for iid sequences, where
E xi xi 0 ui 2
= E xi xi 0 ( yi
= n
1
n
xi 0 b
β )2 = n 1
p
∑ xi xi0 ubi 2 ! Exi xi0 ui2 .
n
∑ xi xi0 (yi
i =1
xi0 b
β )2
(9)
i =1
Thus (7) holds.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
24 / 47
We now consider the Wald test for H0 : R β = r (q linear restrictions):
W
= (R b
β
d
r )0 R (∑i xi xi0 ) 1 (∑i u
bi2 xi xi0 )(∑i xi xi0 ) 1 R 0
! χ2 (q ).
The Wald statistic in the bootstrap world: W =
b
[R ( b
β
β)]0 R (∑i xi xi 0 ) 1 (∑i u
bi 2 xi xi 0 )(∑i xi xi 0 ) 1 R 0
where u
b =y
x 0b
β .
i
i
We can show that
Ke-Li Xu (Indiana University)
i
1
1
(R b
β
R (b
β
r)
b
β ),
d
W ! χ2 (q ).
Bootstrap
(10)
September 13, 2019
25 / 47
p
bi 2 ! Exi xi0 ui2 .
To show (10), we only need to show n1 ∑ni=1 xi xi 0 u
Note that
1 n
bi 2
xi xi 0 u
n i∑
=1
=
1 n
xi xi 0 ( yi
n i∑
=1
=
1 n
xi xi 0 [ u i
n i∑
=1
=
1 n
xi xi 0 ui 2 + op (1)
n i∑
=1
β )2
xi 0 b
β
xi 0 (b
b
β)]2
p
! Exi xi 0 ui 2
p
! Exi xi0 ui2 ,
where the second line uses the bootstrap model yi = xi 0 b
β + ui , and the
1
/
2
b
b
third line uses β
β = Op ( n
) (by (7)), and the last line is by (9).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
26 / 47
Residual-based iid bootstrap
The model remains the same:
yi = xi0 β + ui ,
where xi is k
1, ui
(0, σ2 ) and Exi ui = 0.
Bootstrap model:
β + ui ,
yi = xi0 b
n 1 ∑ni=1 u
bi g (centered residuals)
This is referred to as (…xed-design) residual-based iid bootstrap.
where ui is a random draw from fu
bi
Centered residuals are needed so that E ui = 0. (If there is an intercept,
residuals are always centered)
Bootstrap data: fyi , xi g.
The OLS: b
β = (∑ni=1 xi xi0 ) 1 (∑ni=1 xi yi ).
OLS residuals u
bi = yi
xi0 b
β .
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
27 / 47
In the bootstrap world,
n
= ( ∑ xi xi0 )
b
β
1
i =1
n
Note that
n 1 /2
n
∑
i =1
1
1
n
β + ui ))
( ∑ xi (xi0 b
i =1
i =1
i =1
= b
β + ( ∑ xi xi0 )
i =1
n
n
( ∑ xi yi ) = ( ∑ xi xi0 )
n
( ∑ xi ui )
i =1
d
xi ui ! N (0, plim n 1
n
∑ Var
i =1
d
(xi ui )) = N (0, σ2 Exi xi0 ),
by CLT for inid sequences, where
n 1
n
∑ Var
( xi u i ) = n
1
i =1
= n
1
n
∑ xi xi0 E
i =1
n
∑ xi xi0
i =1
So
Ke-Li Xu (Indiana University)
n 1 /2 ( b
β
ui 2
n 1
n
i =1
d
b
β) ! N (0, σ2 (Exi xi0 ) 1 ).
Bootstrap
p
∑ ubi2 ! σ2 Exi xi0 .
September 13, 2019
28 / 47
Thus, the iid residual-based bootstrap is only valid when conditional
homoskedasticity (CH, i.e. E (ui2 jxi ) = σ2 ) holds.
Under CH, in the original data world
n 1 /2 ( b
β
d
β) ! N (0, σ2 (Exi xi0 ) 1 ).
Without conditional homoskedasticity,
n 1 /2 ( b
β
d
β) ! N (0, (Exi xi0 ) 1 (Exi xi0 ui2 )(Exi xi0 ) 1 ).
b
Thus in genernal, without imposing CH, n1 /2 (b
β
β) does not provide a
1
/
2
b
valid approximation of n ( β β), distribution-wise.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
29 / 47
Wild Bootstrap
Now we consider a di¤erent type of bootstrap
iid
bi ei , where ei
Wild bootstrap: Let ui = u
(0, 1).
Under this scheme, the randomness in the bootstrap world comes from ei
(the original data are considered …xed, as common in the bootstrap analysis).
No need for centered residuals.
Several auxilary variables ei are used in practice:
I
I
Rademacher two-point random variables: P(ep
i = 1)=P(e
i = 1)=1/2.
p
Mammen’s two-point distribution: P(ei = 1 +2 5 )= 5p 1 and
P(ei =
I
1
p
2
Some use ei
5
2 5
p
)= 5p+1 .
2 5
N (0, 1).
Rademacher two-point random variables are recommended for ei .
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
30 / 47
Then
n 1 /2
n
∑ xi ui
= n
d
=
n 1 /2 ( b
β
n
∑ xi ubi ei
i =1
i =1
Thus
1 /2
d
! N (0, plimn
N (0, Exi xi0 ui2 ).
1
n
∑ xi xi0 ubi2 )
i =1
d
b
β) ! N (0, (Exi xi0 ) 1 (Exi xi0 ui2 )(Exi xi0 ) 1 ).
(11)
Now the bootstrap is asymptotically valid.
e.g. con…dence interval for βj can be constructed (using the distribution of
b
n 1 /2 ( b
β
β ) to approximate that of n1 /2 (b
β
β )).
j
j
Ke-Li Xu (Indiana University)
j
Bootstrap
j
September 13, 2019
31 / 47
Recovering the DGP is important, the construction of the statistic is not
Bootstrapping conditional homoskedasticity-based test is still valid under
heteroskedasticiy if wild bootstrap is used.
Consider the case k = 1.
Under original data, the standard t-stat is
(b
β
h
i 1 /2
d
(Ex 2 )
β) σ
! N 0, i
b2 (∑ni=1 xi2 ) 1
1 (Ex 2 u 2 )(Ex 2 )
i i
i
σ2 (Exi2 ) 1
1
.
In bootstrap world (using wild bootstrap),
(b
β
h
i 1 /2
d
(Ex 2 )
b
β) σ
! N 0, i
b 2 (∑ni=1 xi2 ) 1
1 (Ex 2 u 2 )(Ex 2 )
i
i i
σ2 (Exi2 ) 1
1
,
using the result (11), and
b 2=n 1
σ
Ke-Li Xu (Indiana University)
n
∑ ubi 2 = n
i =1
1
n
∑ ubi2 ei2 = n
i =1
Bootstrap
1
n
∑ ubi2 + op
i =1
p
(1) ! σ 2 .
September 13, 2019
32 / 47
Restricted Bootstrap
When performing a test, we can use restricted estimator e
β (under the null
e
hypothesis) in the bootstrap DGP. Thus β is the true value in the bootstrap
universe.
This subsection and the next both highlight the role played by the true value
used in the bootstrap DGP.
We illustrate this idea in the regression model:
yi = xi0 β + ui .
Suppose H0 : β = β0 .
In this case, e
β = β0 .
Generally, we can consider H0 : R β = R 0 . Recall how e
β is obtained in this
case (CLS or EMD).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
33 / 47
Bootstrap DGP:
where ui = u
ei ei , with u
ei = yi
(wild bootstrap).
yi = xi0 β0 + ui ,
xi0 β0 (restricted residual) and ei
Bootstrap data: fyi , xi : i = 1, ..., n g.
Bootstrap test: use the distribution of n1 /2 (b
β
distribution of n1 /2 (b
β β0 ).
iid
(0, 1)
β0 ) to approximate the
There is evidence that the restricted bootstrap is more precise than the
unrestricted bootstrap (which we covered before).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
34 / 47
Simple calculations indicate the validity:
n 1 /2 ( b
β
since
b
β
d
β0 ) ! N (0, (Exi xi0 ) 1 (Exi xi0 ui2 )(Exi xi0 ) 1 ),
n
= ( ∑ xi xi0 )
i =1
1
n
n
( ∑ xi yi ) = ( ∑ xi xi0 )
i =1
i =1
n
n
i =1
i =1
1
(12)
n
( ∑ xi (xi0 β0 + ui ))
i =1
β0 + ( ∑ xi xi0 ) 1 ( ∑ xi ui ),
=
where
n 1 /2
n
∑ xi u i
= n
i =1
1 /2
n
∑ xi uei ei
i =1
d
=
d
! N (0, plimn
N (0, Exi xi0 ui2 ),
1
n
∑ xi xi0 uei2 )
i =1
noting that u
ei = ui under H0 : β = β0 .
The consistency of the test follows from the fact that (12) is true regardless
H0 . (On the other hand, H0 needs to be true so that
d
n 1 /2 ( b
β β0 ) ! N (0, (Exi xi0 ) 1 (Exi xi0 ui2 )(Exi xi0 ) 1 ). Otherwise
n 1 /2 ( b
β β0 ) would diverge.)
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
35 / 47
Bootstrapping F Test
This is an example of bootstrapping criterion function-based tests when the
criterion function involves restrictions.
Consider the regression model:
yi = xi0 β + ui ,
where for simplicity, we assume conditional homoskedasticity.
Suppose H0 : R β = R 0 , where R is q
The F-statistic
F =
k.
e2 σ
b2 )/q
(σ
,
2
b / (n k )
σ
e2 is restricted residual-variance estimator.
where σ
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
36 / 47
Bootstrap sample: fyi , xi g. (e.g. iid residual-based)
The bootstrap F-statistic
F =
e 2 σ
b 2 )/q
(σ
,
b 2 / (n k )
σ
e 2 is calculated using fyi , xi g and imposing the restriction R β = R b
where σ
β.
(A common mistake is still imposing the restriction R β = R 0 ).
[Think about how it works if β = ( β1 , β2 )0 and the null is β2 = 0.]
Then, as always, the distribution of F is used to approximate the
distribution of F .
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
37 / 47
An alternative is to consider the restricted bootstrap
Restricted bootstrap sample: fyi , xi g. (e.g. iid residual-based).
The restricted bootstrap F-statistic
F
=
e 2 σ
b 2 )/q
(σ
,
b 2 / (n k )
σ
e 2 is calculated using fyi , xi g and imposing the restriction R β = R 0 .
where σ
The distribution of F
Ke-Li Xu (Indiana University)
is used to approximate the distribution of F .
Bootstrap
September 13, 2019
38 / 47
Final words: Parametric Bootstrap
What we have discussed is called nonparametric bootstrap, since the
empirical CDF Fb is a nonparametric estimator of F .
Nonparametric bootstrap is what most applications in econometrics use.
Parametric bootstrap: utilizes the function form of F .
Suppose yi
F (y j β), where F has a known form but β is unknown.
F (y j b
β), where b
β is the maximum
Parametric bootstrap draws yi
likelihood (ML) estimator of β.
Denote b
β is the MLE using the bootstrap data fy : i = 1, ..., n g.
i
We then use use the distribution of
distribution of n1 /2 (b
β β ).
Ke-Li Xu (Indiana University)
n 1 /2 ( b
β
Bootstrap
b
β) to approximate the
September 13, 2019
39 / 47
The second example: Gaussian linear regression model, yi jxi
b2
MLE: b
β, σ
Parametric bootstrap:
N (xi0 β, σ2 ).
β + ui ,
yi = xi0 b
b 2 ).
where ui
N (0, σ
Think about how this is di¤erent from the wild bootstrap when we use the
auxilary variable ei
N (0, 1).
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
40 / 47
A permutation test
Consider the single-regressor linear regression
yi = β 0 + β 1 xi + u i .
(13)
We test H0 : β1 = 0.
While we can use the asymptotic approach or the bootstrap, a permutation
test can also be used.
The idea is that if β1 = 0, then the order of fxi : i = 1, ..., n g shouldn’t
matter, if the order of fyi : i = 1, ..., n g is kept unchanged.
Let π (1), ..., π (n ) be a permutation of (1, ..., n ).
Suppose all permutations we consider are in the set Π. Then jΠj
n!.
For each permutation π, compute the least squares estimator of β1 using the
π
data fxπ (i ) , yi g, denoted as b
β1 .
π
If β1 = 0, b
β1 and b
β1 (original estimator) should come from the same
distribution.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
41 / 47
Here the test statistic is b
β1 .
π
We approximate its distribution Gn by GnΠ (the empirical distribution b
β1 over
fπ 2 Πg).
Denote the α-th quantile of GnΠ by q Π (α).
We reject H0 at level 5% if
b
β1 < q Π (0.025).
β1 > q Π (0.975) or b
If the approximation of Gn by GnΠ is valid, then under the null,
P (b
β1 2rejection region) = 0.05.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
42 / 47
Compare with bootstrap.
The bootstrap resamples the pair fxi , yi g (without re-matching within the
pair).
The permutation test resamples xi (without replacement), while using the
same data (in the same order) yi for each permutation.
In general, if you are interested in H0 : β1 = β01 , rewrite the model as
yi
β01 xi = β0 + ( β1
β01 )xi + ui .
Then the permutation test is implemented the same as above except using
the outcome yi β01 xi .
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
43 / 47
Asymptotics of the permutation test
Although such a test is widely used, it is only asymptotically valid under
conditional homoskedasticity (i.e. E (ui2 jxi ) = E (ui2 ) = σ2 ).
Asymptotic validity here means Gn and GnΠ converge to the same
distribution.
An important implication for CH is: for any π 2 Π, for each i,
E [ui2 (xπ (i )
Ex )2 ] = σ2 Var (x ).
(14)
It is because
=
E [ui2 (xπ (i )
(
E [ui2 (xi
Ex )2 ]
CH
Ex )2 ] = σ2 Var (x ),
Eui2 E (xπ (i )
iid data
Ex )2 =
σ2 Var (x ),
if π (i ) = i;
if π (i ) 6= i.
We can show that under H0 ,
Ke-Li Xu (Indiana University)
π d
n 1 /2 b
β1 ! N (0, σ2 Var (x ) 1 ).
Bootstrap
(15)
September 13, 2019
44 / 47
To prove (15), by the FWL theorem, under H0 ,
π
b
β1
=
h
∑ni=1 (xπ (i )
x )2
h
= ∑ni=1 (xπ (i )
H0
=
∑ni=1 (xi
i 1 n
∑ ( xπ (i )
x )2
x )(yi
i =1
i 1 n
∑ ( xπ (i )
y)
x ) ui
i =1
1
x )2
n
∑ ( xπ (i )
x ) ui .
i =1
We then have
π d
β1 ! N (0, Vπ ),
n 1 /2 b
where
Vπ
= Var (xi )
2
lim n 1
(16)
n
∑ Eui2 (xπ(i )
Exi )2
(17)
i =1
(14 )
2
= Var (xi )
= σ2 Var (x )
Ke-Li Xu (Indiana University)
1
Eui2 (xπ (i )
Exi )2
(18)
.
Bootstrap
September 13, 2019
45 / 47
In (16) above, we have used the CLT for independent but not identically
distributed data.
E.g. Suppose π : (1, 2, 3) ! (3, 2, 1). Then (x3 Ex )u1 and (x2 Ex )u2
are not identically distributed, if there is (higher order) dependece between x2
and u2 . These two are independent (by considering the correlation of any
moments of these two).
Thus (15) holds.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
46 / 47
If we allow conditional heteroskedasticity, Vπ takes the general form of (17).
(instead of (18))
It is because Eui2 (xπ (i ) Exi )2 may di¤er across i for a particular π.
This happens if π does not move every unit.
I
I
For i such that π (i ) 6= i , Eui2 (xπ (i ) Ex )2 = Eui2 E (xπ (i ) Ex )2 = σ2 Var (x ).
But for i such that π (i ) = i , Eui2 (xπ (i ) Ex )2 6= σ2 Var (x ).
If π moves every unit (like π : (1, 2, 3, 4) ! (2, 3, 4, 1)), then
Vπ = σ2 Var (x ) 1 .
π
Thus Vπ depends on π. We thus don’t expect the distribution b
β1 over Π
would provide a useful approximation.
Ke-Li Xu (Indiana University)
Bootstrap
September 13, 2019
47 / 47