Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Foundations of statistics wikipedia , lookup
History of statistics wikipedia , lookup
Taylor's law wikipedia , lookup
Statistical inference wikipedia , lookup
German tank problem wikipedia , lookup
Misuse of statistics wikipedia , lookup
Student's t-test wikipedia , lookup
Bootstrapping A nonparametric Approach to Statistical Inference Mooney & Duval Series on Quantitative Applications in the Social Sciences, 95, Sage University Papers Seminar General Statistics, 25 October 2012 Wendy Post Statistician Department of Special Education University of Groningen 1 overview • • • • • • • Introduction: what is bootstrapping Introduction of a real dataset Estimation of standard errors Confidence intervals Estimation of bias Testing Literature 2 1 Quantitative research Real world 3 theoretical world scientific models ϴ Data statistical models Conclusions Based on R.E. Kass (2011): the big picture Bootstrap method Quantitative research: testing and estimation require information about the population distribution. What if the usual assumptions of the population cannot be made (normality assumptions, or small samples): the traditional approach does not work Bootstrap: Estimates the population distribution by using the information based on a number of resamples from the sample. 2 Bootstrap Wikipedia: Boots may have a tab, loop or handle at the top known as a bootstrap, allowing one to use fingers or a tool to provide greater force in pulling the boots on. Bootstrap method Impossible task: to pull oneself up by one's bootstraps A metaphor for ‘a self-sustaining process that proceeds without external help’. 3 Bootstrap method Use the information of a number of resamples from the sample to estimate the population distribution Given a sample of size n: Procedure • treat the sample as population • Draw B samples of size n with replacement from your sample (the bootstrap samples) • Compute for each bootstrap sample the statistic of interest (for example, the mean) • Estimate the sample distribution of the statistic by the bootstrap sample distribution Bootstrap method 8 Real world bootstrap world unknown probability distribution/model Observed data estimated probability model Bootstrap sample P X = (x1,x2,…,xn) P̂ X* = (x1*,x2*,…,xn*) Statistic (sample mean) Statistic Bootstrap sample (mean of X*) Based on Efron and Tibshirani (1993) 4 Example Population: standard normal distribution Sample with sample size = 30. 1000 bootstrap samples. Results sample (traditional analysis) Sample mean: 0.1698 95% Confidence interval: (-0.12; 0.46) Bootstrap results: sample mean: 0.1689; 95% Confidence interval : (-0.10, 0.45) Example Case number Original data B_sample 1 B_sample 2 B_sample 3 1 2 3 4 5 6 7 8 9 . . 25 26 27 28 29 30 1.22 -0.49 0.61 0.72 0.58 -0.08 0.07 -0.26 -0.07 -0.49 -0.93 0.72 1.22 -0.49 -0.49 1.22 -0.17 0.40 -0.17 -0.41 1.40 1.27 0.26 -0.55 1.27 -0.52 -1.20 -0.15 0.26 -0.17 1.30 -1.20 -0.26 -0.08 0.40 -0.15 -0.55 1.22 0.11 . . 0.52 1.40 1.30 -0.55 0.07 1.40 1.40 -0.07 -0.55 0.13 1.27 -0.08 -0.08 0.61 -0.15 . . 0.15 0.72 1.22 0.07 0.40 0.13 Mean Sd 0.17 0.81 0.20 0.73 1.11 0.83 0.32 0.85 5 R-script Based on Crawley: The R book, p.320 set.seed(146035) #start seed for simulation sampleW <- rnorm(30, 0, 1) # draw sample of size 30 meansW <- numeric(1000) #declaration of de bootstrap mean vector # bootstrap procedure for (i in 1:1000){ meansW[i] <- mean(sample(sampleW, replace = T))} R-script # bootstrap results hist(meansW,30) summary(meansW) quantile(meansW, c(0.025, 0.975)) # traditional approach summary(sampleW) mean(sampleW) CI_up = mean(sampleW)+1.96*sd(sampleW)/sqrt(30) CI_lo = mean(sampleW)-1.96*sd(sampleW)/sqrt(30) 6 Bootstrap results Results example Population: standard normal distribution Sample with sample size = 30. 1000 bootstrap samples. Results sample Sample mean: 0.1698 95% Confidence interval: (-0.12; 0.46) Bootstrap results: sample mean: 0.1689; 95% Confidence interval : (-0.10, 0.45) 7 R-script : alternative Based on Crawley: The R book, p.321 library(boot) set.seed(146035) #start seed for simulation sampleW <- rnorm(30, 0, 1) # draw sample of size 30 mymean= function(sampleW, i) mean(sampleW[i]) myboot = boot(sampleW, mymean, R=1000) myboot R-script : output ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = sampleW, statistic = mymean, R = 1000) Bootstrap Statistics : original t1* 0.1698585 bias 7.697321e-05 std. error 0.1484104 8 Why does bootstrapping work? Basic idea: If the sample is a good approximation of the population, bootstrapping will provide a good approximation of the sample distribution. Justification: 1. If the sample is representative for the population, the sample distribution (empirical distribution) approaches the population (theoretical) distribution if n increases. 2. If the number of resamples (B) from the original sample increases, the bootstrap distribution approaches the sample distribution bootstrapping • Estimation of the standard error of your statistic of interest • Construction of confidence intervals • Estimation of bias • Bootstrap testing (for small samples, closely related to permutation tests) 9 Comparison of two independent groups Buddy project (De Boer) Study aim: To determine the effect of a buddy program intervention for influencing the attitude of typically developing students towards children with severe intellectual and/or physical disabilities Comparison of two groups: Buddy intervention: 4 typically developing students each being a buddy for a child with disabilities Control group: 4 typically developing students without being a buddy for a child with disabilities Dependent variable: difference in attitude(post-pre), measured by Attitudes Survey towards Inclusive Education (ASIE) 19 Manipulated data for the Buddy project Research question: What is the effect of the buddy intervention on the attitude compared to the control group? 20 10 Manipulated data Data Buddy project 21 Manipulated data Buddy project Design: two independent groups Test statistic: difference in means = 6.25 22 11 manipulated data Buddy project How to bootstrap for estimation of the standard error of the sample statistic? General 1. Select B independent bootstrap samples (replications): sample with replacement: x1*,…x8* B times (for estimation of standard errors, B = 25-200 is OK) 2. Compute the bootstrap sample statistic for each replication, θ*(b), b = 1,…B 3. Determine the bootstrap sample distribution of the bootstrap sample statistic 4. Estimate the standard deviation of the bootstrap sample statistic (standard error of the estimate) 23 manipulated data Buddy project In our Buddy project: 1. Select B independent bootstrap samples (replications) with replacement from the data of the children in the buddy intervention x1*,…x4* 2. Select B independent bootstrap samples (replications) with replacement from the data of the children in the control intervention x5*,…x8* 3. Compute the difference in means in both interventions for each bootstrap sample; difmean*(b), b = 1,…B 4. Determine the bootstrap sample distribution of difmean* 4. Estimate the standard deviation of the difmean* (standard error) We will do this for B = 25, 200, 1000 and 10,000 24 12 Frequency distribution of the bootstrap sample statistic (difmean*) B= 25 Histogram of ms B= 200 30 Frequency 0 0 10 2 20 4 Frequency 40 6 50 8 60 Histogram of ms -2 -2 0 2 4 6 8 10 0 2 4 6 8 10 12 12 ms ms Histogram of ms Histogram of ms 1500 B= 10000 0 0 50 500 Frequency F re q u e n c y 10 0 1000 15 0 B=1000 0 0 5 5 10 10 ms 25 ms Buddy project statistics of the bootstrap distribution of difmean* Number of Mean Median replications 25 200 1000 10000 5.88 6.24 6.33 6.26 6.0 6.5 6.5 6.25 the standard error sd 3.11 2.46 2.48 2.47 26 13 Buddy project percentiles All percentiles can be computed: estimation of the confidence intervals: percentile method Number of 2.5% replications sd 50% 97.5% 25 200 1000 10000 6.0 6.5 6.5 6.25 10.25 10.5 10.5 10.5 0.30 1.24 1.0 1.0 lower boundary upper boundary 27 bootstrap for bias estimation Bias of an estimator: difference between population value θ and the expected value of that estimator . In formula: bias = θ - E( ) So is the estimator on average correct? If the theoretical assumptions about distributions are not satisfied, then the estimators may be biased 28 14 bootstrap for bias estimation The difference between original sample statistic and the bootstrap sample statistic can be regarded as the estimation of the bias. Sample mean of difference was: 6.25 Estimated bias = bootstrap mean – sample mean B Mean θ* Bias θ*- 25 200 1000 10000 5.88 6.24 6.33 6.26 -0.37 -0.01 0.08 0.01 corrected mean 6.62 6.26 6.17 6.24 29 Buddy project: Testing the difference between two interventions 1. Draw B independent bootstrap samples (replications): sample with replacement: x1*,…x8*. 2. Assign the first 4 observations x1*,…, x4* to the buddy project, and the last 4 observations, x4*,…,x8* to the control group. Note: this is the distribution under the null hypothesis of equal distributions. 3. Compute for each the bootstrap sample the difference in means, θ*(b), b = 1,…B 4. Approximate the p-value by the number of bootstrap sample differences larger than the original sample mean (e.g. 6.25) 30 15 Buddy project: Testing the difference between two interventions Results • 282 cases of 10.000 gives a sample mean difference > 6.25 • One sided P-value: 0.0282 • 287 cases of 10.000 gives sample value < -6.25 • two-sided p-value: 0.057 31 sample size Sample size: Bootstrapping is working well for samples >30-50 (if the sampling procedure is random) For smaller sample sizes: there might be problems about accuracy and validity: be careful with the interpretation. Number of bootstrap samples: The number of bootstrap samples depends on what you like to do. For estimation of standard error: 25-200 resamples should be sufficient. For all other applications: Take B> 1000. 32 16 Comments about bootstrapping • Only the basic principles are introduced here • The sampling method depends on the stochastic element one would like to generate (depends on the model!). Only the simplest cases are demonstrated here. • The strength of bootstrapping lies in estimation rather than in testing (certainly for small samples) • The validity of the bootstrap testing is questionable for small sample sizes 33 Comments about bootstrapping (cont) • There are more advanced ways to construct confidence intervals with bootstrapping – BCα-method (bias-correction and accelerated) – ABC-method (approximate confidence interval) • Testing with the bootstrap method in small samples is closely connected with the randomization tests: randomization tests are the exact versions. • Bootstrap can be applied to more complex designs and in more situations (there are several ways in regression models) 34 17 literature Moony, C.Z. and Duval, R.D. (1993). Bootstrapping. A nonparametric approach to statistical inference. London: Sage publications. Efron, B. and Tibshirani, R.J. (1993). An introduction to the bootstrap. London: Chapman & Hall. Ter Braak, C.J.F. (1992). Permutation versus bootstrap significance tests in multiple regression and anova. In Bootstrapping and related techniques, K.H.Jockel, G.Rothe & W. Sendler, (eds.), 79-86. R.E. Kass (2011). Statistical inference: The big picture, Statistical Science 26(1) 1-9. 35 18