Download 2012 10 25 Bootstrap

Bootstrapping
A nonparametric Approach to Statistical Inference
Mooney & Duval
Series on Quantitative Applications in the Social
Sciences, 95, Sage University Papers
Seminar General Statistics, 25 October 2012
Wendy Post
Statistician
Department of Special Education
University of Groningen
1
overview
•
•
•
•
•
•
•
Introduction: what is bootstrapping
Introduction of a real dataset
Estimation of standard errors
Confidence intervals
Estimation of bias
Testing
Literature
2
1
Quantitative research
Real world
3
theoretical world
scientific models
ϴ
Data
statistical models
Conclusions
Based on
R.E. Kass (2011):
the big picture
Bootstrap method
Quantitative research: testing and estimation require
information about the population distribution.
What if the usual assumptions of the population cannot be
made (normality assumptions, or small samples): the
traditional approach does not work
Bootstrap:
Estimates the population distribution by using the
information based on a number of resamples from the
sample.
2
Bootstrap
Wikipedia:
Boots may have a tab, loop or handle at the top
known as a bootstrap, allowing one to use fingers or a
tool to provide greater force in pulling the boots on.
Bootstrap method
Impossible task:
to pull oneself up by one's bootstraps
A metaphor for
‘a self-sustaining process that
proceeds without external help’.
3
Bootstrap method
Use the information of a number of resamples from the
sample to estimate the population distribution
Given a sample of size n:
Procedure
• treat the sample as population
• Draw B samples of size n with replacement from your
sample (the bootstrap samples)
• Compute for each bootstrap sample the statistic of
interest (for example, the mean)
• Estimate the sample distribution of the statistic by the
bootstrap sample distribution
Bootstrap method
8
Real world
bootstrap world
unknown
probability
distribution/model
Observed data
estimated
probability
model
Bootstrap sample
P
X = (x1,x2,…,xn)
P̂
X* = (x1*,x2*,…,xn*)
Statistic
(sample mean)
Statistic
Bootstrap sample
(mean of X*)
Based on Efron and Tibshirani (1993)
4
Example
Population: standard normal distribution
Sample with sample size = 30.
1000 bootstrap samples.
Results sample (traditional analysis)
Sample mean: 0.1698
95% Confidence interval: (-0.12; 0.46)
Bootstrap results:
sample mean: 0.1689;
95% Confidence interval : (-0.10, 0.45)
Example
Case number
Original data
B_sample 1
B_sample 2
B_sample 3
1
2
3
4
5
6
7
8
9
.
.
25
26
27
28
29
30
1.22
-0.49
0.61
0.72
0.58
-0.08
0.07
-0.26
-0.07
-0.49
-0.93
0.72
1.22
-0.49
-0.49
1.22
-0.17
0.40
-0.17
-0.41
1.40
1.27
0.26
-0.55
1.27
-0.52
-1.20
-0.15
0.26
-0.17
1.30
-1.20
-0.26
-0.08
0.40
-0.15
-0.55
1.22
0.11
.
.
0.52
1.40
1.30
-0.55
0.07
1.40
1.40
-0.07
-0.55
0.13
1.27
-0.08
-0.08
0.61
-0.15
.
.
0.15
0.72
1.22
0.07
0.40
0.13
Mean
Sd
0.17
0.81
0.20
0.73
1.11
0.83
0.32
0.85
5
R-script
Based on Crawley: The R book, p.320
set.seed(146035) #start seed for simulation
sampleW <- rnorm(30, 0, 1) # draw sample of size 30
meansW <- numeric(1000) #declaration of de bootstrap mean vector
# bootstrap procedure
for (i in 1:1000){
meansW[i] <- mean(sample(sampleW, replace = T))}
R-script
# bootstrap results
hist(meansW,30)
summary(meansW)
quantile(meansW, c(0.025, 0.975))
# traditional approach
summary(sampleW)
mean(sampleW)
CI_up = mean(sampleW)+1.96*sd(sampleW)/sqrt(30)
CI_lo = mean(sampleW)-1.96*sd(sampleW)/sqrt(30)
6
Bootstrap results
Results example
Population: standard normal distribution
Sample with sample size = 30.
1000 bootstrap samples.
Results sample
Sample mean: 0.1698
95% Confidence interval: (-0.12; 0.46)
Bootstrap results:
sample mean: 0.1689;
95% Confidence interval : (-0.10, 0.45)
7
R-script : alternative
Based on Crawley: The R book, p.321
library(boot)
set.seed(146035) #start seed for simulation
sampleW <- rnorm(30, 0, 1) # draw sample of size 30
mymean= function(sampleW, i) mean(sampleW[i])
myboot = boot(sampleW, mymean, R=1000)
myboot
R-script : output
ORDINARY NONPARAMETRIC BOOTSTRAP
Call:
boot(data = sampleW, statistic = mymean, R = 1000)
Bootstrap Statistics :
original
t1*
0.1698585
bias
7.697321e-05
std. error
0.1484104
8
Why does bootstrapping work?
Basic idea:
If the sample is a good approximation of the population,
bootstrapping will provide a good approximation of the
sample distribution.
Justification:
1. If the sample is representative for the population, the
sample distribution (empirical distribution) approaches
the population (theoretical) distribution if n increases.
2. If the number of resamples (B) from the original sample
increases, the bootstrap distribution approaches the
sample distribution
bootstrapping
• Estimation of the standard error of your statistic
of interest
• Construction of confidence intervals
• Estimation of bias
• Bootstrap testing (for small samples, closely
related to permutation tests)
9
Comparison of two independent groups
Buddy project (De Boer)
Study aim:
To determine the effect of a buddy program intervention for
influencing the attitude of typically developing students towards
children with severe intellectual and/or physical disabilities
Comparison of two groups:
Buddy intervention: 4 typically developing students each being a
buddy for a child with disabilities
Control group: 4 typically developing students without being a
buddy for a child with disabilities
Dependent variable: difference in attitude(post-pre), measured
by Attitudes Survey towards Inclusive Education (ASIE)
19
Manipulated data for the Buddy project
Research question:
What is the effect of the buddy intervention on the attitude
compared to the control group?
20
10
Manipulated data
Data Buddy project
21
Manipulated data Buddy project
Design: two independent groups
Test statistic: difference in means = 6.25
22
11
manipulated data Buddy project
How to bootstrap for estimation of the standard error of the
sample statistic?
General
1. Select B independent bootstrap samples (replications):
sample with replacement: x1*,…x8* B times
(for estimation of standard errors, B = 25-200 is OK)
2. Compute the bootstrap sample statistic for each replication,
θ*(b), b = 1,…B
3. Determine the bootstrap sample distribution of the
bootstrap sample statistic
4. Estimate the standard deviation of the bootstrap sample
statistic (standard error of the estimate)
23
manipulated data Buddy project
In our Buddy project:
1. Select B independent bootstrap samples (replications) with
replacement from the data of the children in the buddy
intervention x1*,…x4*
2. Select B independent bootstrap samples (replications) with
replacement from the data of the children in the control
intervention x5*,…x8*
3. Compute the difference in means in both interventions for
each bootstrap sample; difmean*(b), b = 1,…B
4. Determine the bootstrap sample distribution of difmean*
4. Estimate the standard deviation of the difmean*
(standard error)
We will do this for B = 25, 200, 1000 and 10,000
24
12
Frequency distribution
of the bootstrap sample statistic (difmean*)
B= 25
Histogram of ms
B= 200
30
Frequency
0
0
10
2
20
4
Frequency
40
6
50
8
60
Histogram of ms
-2
-2
0
2
4
6
8
10
0
2
4
6
8
10
12
12
ms
ms
Histogram of ms
Histogram of ms
1500
B= 10000
0
0
50
500
Frequency
F re q u e n c y
10 0
1000
15 0
B=1000
0
0
5
5
10
10
ms
25
ms
Buddy project
statistics of the bootstrap distribution of
difmean*
Number of Mean Median
replications
25
200
1000
10000
5.88
6.24
6.33
6.26
6.0
6.5
6.5
6.25
the standard error
sd
3.11
2.46
2.48
2.47
26
13
Buddy project
percentiles
All percentiles can be computed: estimation of
the confidence intervals: percentile method
Number of 2.5%
replications sd
50%
97.5%
25
200
1000
10000
6.0
6.5
6.5
6.25
10.25
10.5
10.5
10.5
0.30
1.24
1.0
1.0
lower boundary
upper boundary
27
bootstrap for bias estimation
Bias of an estimator: difference between
population value θ and the expected value of
that estimator .
In formula: bias = θ - E( )
So is the estimator on average correct?
If the theoretical assumptions about
distributions are not satisfied, then the
estimators may be biased
28
14
bootstrap for bias estimation
The difference between original sample statistic and the
bootstrap sample statistic can be regarded as the
estimation of the bias.
Sample mean of difference was: 6.25
Estimated bias = bootstrap mean – sample mean
B
Mean
θ*
Bias
θ*-
25
200
1000
10000
5.88
6.24
6.33
6.26
-0.37
-0.01
0.08
0.01
corrected
mean
6.62
6.26
6.17
6.24
29
Buddy project:
Testing the difference between two interventions
1. Draw B independent bootstrap samples (replications):
sample with replacement: x1*,…x8*.
2. Assign the first 4 observations x1*,…, x4* to the buddy
project, and the last 4 observations, x4*,…,x8* to the control
group. Note: this is the distribution under the null
hypothesis of equal distributions.
3. Compute for each the bootstrap sample the difference in
means, θ*(b), b = 1,…B
4. Approximate the p-value by the number of bootstrap sample
differences larger than the original sample mean (e.g. 6.25)
30
15
Buddy project:
Testing the difference between two interventions
Results
• 282 cases of 10.000 gives
a sample mean difference
> 6.25
• One sided P-value: 0.0282
• 287 cases of 10.000 gives
sample value < -6.25
• two-sided p-value: 0.057
31
sample size
Sample size:
Bootstrapping is working well for samples >30-50 (if the
sampling procedure is random)
For smaller sample sizes: there might be problems about
accuracy and validity: be careful with the interpretation.
Number of bootstrap samples:
The number of bootstrap samples depends on what you like to
do.
For estimation of standard error: 25-200 resamples should be
sufficient.
For all other applications: Take B> 1000.
32
16
Comments about bootstrapping
• Only the basic principles are introduced here
• The sampling method depends on the stochastic element one
would like to generate (depends on the model!). Only the
simplest cases are demonstrated here.
• The strength of bootstrapping lies in estimation rather than in
testing (certainly for small samples)
• The validity of the bootstrap testing is questionable for small
sample sizes
33
Comments about bootstrapping (cont)
• There are more advanced ways to construct confidence
intervals with bootstrapping
– BCα-method (bias-correction and accelerated)
– ABC-method (approximate confidence interval)
• Testing with the bootstrap method in small samples is closely
connected with the randomization tests: randomization tests
are the exact versions.
• Bootstrap can be applied to more complex designs and in
more situations (there are several ways in regression models)
34
17
literature
Moony, C.Z. and Duval, R.D. (1993). Bootstrapping. A nonparametric
approach to statistical inference. London: Sage publications.
Efron, B. and Tibshirani, R.J. (1993). An introduction to the bootstrap.
London: Chapman & Hall.
Ter Braak, C.J.F. (1992). Permutation versus bootstrap significance tests in
multiple regression and anova. In Bootstrapping and related techniques,
K.H.Jockel, G.Rothe & W. Sendler, (eds.), 79-86.
R.E. Kass (2011). Statistical inference: The big picture, Statistical Science
26(1) 1-9.
35
18