Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Introduction to resampling methods
Transcript
overview Randomisation tests Cross-validation Jackknife Bootstrap Introduction to resampling methods Vivien Rossi CIRAD - UMR Ecologie des forêts de Guyane [email protected] Master 2 - Ecologie des Forêts Tropicale AgroParisTech - Université Antilles-Guyane Kourou, novembre 2010 Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife objectives of the course 1 to present resampling technics randomization tests cross-validation jackknife bootstrap 2 to apply with R Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife outlines 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion Resampling in statistics Description set of statistical inference methods based on new samples drawn from the initial sample Implementation computer simulation of these new samples analysing these new data to refine the inference Classical uses estimation/bias reduction of an estimate (jackknife, bootstrap) estimation of confidence intervalle without normality assumption (bootstrap) exacts tests (permutation tests) model validation (cross validation) Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife History of resampling techniques 1935 randomization tests, Fisher 1948 cross-validation, Kurtz 1958 jackknife, Quenouille-Tukey 1979 bootstrap, Efron Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion Resampling mecanism Why it works ? can we expect an improvement by resampling from the same sample? no new information is brought back ! but it can help to extract useful information from the base sample the idea to consider the sample like the population to simulate samples that we could see to handle "scale" relationship into the sample Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap Illustration with Russian dolls How many flowers is there on the biggest doll? ↑ population ↑ sample Vivien Rossi | {z sub-samples resampling } Conclusion overview Randomisation tests Cross-validation Jackknife 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion randomization tests or permutation tests goal: testing assumption H0 : X and Y are independants H1 : X and Y are dependants principle: data are randomly re-assigned so that a p-value is calculated based on the permutated data permutation tests exhaust all possible outcomes ⇒ exact tests ⇒ exact p-value randomization tests resampling simulates a large number of possible outcomes ⇒ approximate p-value Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap example from Fisher: Lady tasting tea a typical british story In 1920, R. A. Fisher met a lady who insisted that her tongue was sensitive enough to detect a subtle difference between a cup of tea with the milk being poured first and a cup of tea with the milk being added later. Fisher was skeptical . . . Fisher experiment: he presented 8 cups of tea to this lady 4 cups were ’milk-first’ and 4 others were ’tea-first’: tea or milk first ? Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife example from Fisher: Lady tasting tea tea or milk first ? experiment result: the lady well detected the 8 cups Did the woman really have a super-sensitive tongue? Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example from Fisher: Lady tasting tea tea or milk first ? experiment result: the lady well detected the 8 cups Did the woman really have a super-sensitive tongue? Reformulation as a statistical test H0 : the order in which milk or tea is poured in a cup and the lady’s detection of the order are independent. H1 : the lady can correctly tell the order in which milk or tea is poured in a cup. Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example from Fisher: Lady tasting tea Reformulation as a statistical test H0 : the order in which milk or tea is poured in a cup and the lady’s detection of the order are independent. H1 : the lady can correctly tell the order in which milk or tea is poured in a cup. Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example from Fisher: Lady tasting tea Reformulation as a statistical test H0 : the order in which milk or tea is poured in a cup and the lady’s detection of the order are independent. H1 : the lady can correctly tell the order in which milk or tea is poured in a cup. probabilities of all the possibilities under H0 (1,0,1,1,0,0,1,0) (1,1,1,1,0,0,0,0) (1,1,1,0,1,0,0,0) .. . number of match 6 4 .. . probability 1/nb possibilities 1/nb possibilities .. . (1,0,1,1,0,0,1,0) .. . 8 .. . 1/nb possibilities .. . (0,0,0,0,1,1,1,1) 2 1/nb possibilities Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion example from Fisher: Lady tasting tea Reformulation as a statistical test H0 : the order in which milk or tea is poured in a cup and the lady’s detection of the order are independent. H1 : the lady can correctly tell the order in which milk or tea is poured in a cup. test result The probability of matching the 8 cups under H0 is 1/(nb possibilities) 8 number of possibilities = 70 4 i.e.The probability that the Lady matched by chance the 8 cups is 1/70 ≈ 0.014 Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion example from Fisher: Lady tasting tea Reformulation as a statistical test H0 : the order in which milk or tea is poured in a cup and the lady’s detection of the order are independent. H1 : the lady can correctly tell the order in which milk or tea is poured in a cup. test result The probability of matching the 8 cups under H0 is 1/(nb possibilities) 8 number of possibilities = 70 4 i.e.The probability that the Lady matched by chance the 8 cups is 1/70 ≈ 0.014 exercise What can we say if the Lady had matched only 6 cups ? Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap example from Fisher: Lady tasting tea exercise What can we say if the Lady had matched only 6 cups ? some tips to do it with R exhausting all possibilities: see function combn matching of all the possibilities: loops for or function apply Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example from Fisher: Lady tasting tea exercise What can we say if the Lady had matched only 6 cups ? some tips to do it with R exhausting all possibilities: see function combn matching of all the possibilities: loops for or function apply a solution TeaCup <- function(NbCup=8){ NbMilk=NbCup/2 ref=rep(0,NbCup) ref[sample(NbCup,NbMilk)]=1 possibilities=combn(1:NbCup,NbMilk) score=rep(0,ncol(possibilities)) for (i in 1:ncol(possibilities)){ score[i]=sum(ref[possibilities[,i]])*2 } Probabilities=table(score)/choose(NbCup,NbMilk) return(Probabilities) } Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: Comparison of two groups Hot-dog data type meat calories 186 181 176 149 184 190 158 139 175 148 152 111 141 153 190 157 131 149 135 132 poultry 129 132 102 106 94 102 87 99 107 113 135 142 86 143 152 146 144 Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: Comparison of two groups Hot-dog data type meat calories 186 181 176 149 184 190 158 139 175 148 152 111 141 153 190 157 131 149 135 132 poultry 129 132 102 106 94 102 87 99 107 113 135 142 86 143 152 146 144 Hypothesis test H0 : the hot-dog type and calories are independents H1 : the hot-dog type and calories are dependants Exercise with R Make the test, functions from package coin may be helpful What test is it if you use the ranks ? Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: test significance of covariate in linear regression Linear model Yi = β0 + β1 xi1 + · · · + βp xip + εi Vivien Rossi resampling i = 1, . . . , n Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: test significance of covariate in linear regression Linear model Yi = β0 + β1 xi1 + · · · + βp xip + εi i = 1, . . . , n hypothesis test H0 : Y and (X1 , . . . , Xp ) are linearly independents H1 : Y and (X1 , . . . , Xp ) are linearly dependents Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: test significance of covariate in linear regression Linear model Yi = β0 + β1 xi1 + · · · + βp xip + εi i = 1, . . . , n hypothesis test H0 : Y and (X1 , . . . , Xp ) are linearly independents H1 : Y and (X1 , . . . , Xp ) are linearly dependents exercise with R simulate data according to the model proceed a permutation test with the function lmp from the package lmPerm Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion Cross-validation goal to estimate how accurately a predictive model will perform in practice principle 1 splits the dataset into training and validation data 2 fit model on training data 3 assess predictive accuracy on validation data (RMSE) some types of cross-validation random subsampling (randomly splits the data into training and validation data) ×s K fold randomly splits the data into K subsamples ( K − 1 for training and 1 for validation)×K leave-one-out (N − 1 obs for training and 1 obs for validation)×N ... Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Example: K-fold data: (xi , yi )i=1,...,100 5 folds: d1 = (xi , yi )i=1,...,20 , d2 = (xi , yi )i=21,...,40 , d3 = (xi , yi )i=41,...,60 , d4 = (xi , yi )i=61,...,80 and d5 = (xi , yi )i=81,...,100 CV for model M: ŷ = f (x, θ) assess prediction for subsample d1 1 2 3 P 2 training: θ1 = arg minθ 100 i=21 (f (xi , θ) − yi ) 1 predicting: ŷi = f (xi ,P θ ) for i = 1, . . . , 20 2 validation: MSE1 = 20 i=1 (ŷi − yi ) /20 idem for d2 , . . . , d5 → MSE2 , . . . , MSE5 CV = mean(MSE1 , MSE2 , MSE3 , MSE4 , MSE5 ) Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap excercise with R linear regression 1 simulate data according to the model y = β0 + β1 x1 + · · · + βk xk + ε 2 compute CV for several covariate combination (use function cv.glm form package boot Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap excercise with R linear regression 1 simulate data according to the model y = β0 + β1 x1 + · · · + βk xk + ε 2 compute CV for several covariate combination (use function cv.glm form package boot generalised linear model 1 simulate data according to the model y ∼ P exp(β0 + β1 x1 + · · · + βk xk ) 2 compute CV for several covariate combination (use function cv.glm form package boot Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap excercise with R linear regression 1 simulate data according to the model y = β0 + β1 x1 + · · · + βk xk + ε 2 compute CV for several covariate combination (use function cv.glm form package boot generalised linear model 1 simulate data according to the model y ∼ P exp(β0 + β1 x1 + · · · + βk xk ) 2 compute CV for several covariate combination (use function cv.glm form package boot any model see function crossval from package boostrap make your own function Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion Cross-validation caracteristics generalities easy to implement, the prediction method need only be available as a "black box" provides ECV a nearly unbiased estimate of the expectation of the fitting criterion can be very slow since the training must be carried out repeatedly, methods to speed up exists but . . . limitations validation set and test set must drawn from the same population be careful with dynamical systems and stratified population instability for small sample variance of CV can be large → to take into account for model selection Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion the Jackknife overview proposed to reduce the bias of an estimator (delete if bias = a/n) generally used to estimate the variance or covariance useless for unbiased estimators (mean) close to leave one out method only adapted for estimators which are smooth function of the observation principle let a sample x1 , . . . , xn ∼ L(θ) let T an estimator of θ computed with x1 , . . . , xn for i = 1, . . . , n: T−i is the estimator computed with x1 , . . . , xi−1 , xi+1 , . . . , xn Pn the Jackknife estimators: TJ = T − n−1 i=1 (T−i − T ) n Pn n−1 bias of the estimator: − n i=1 (T−i − T ) Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap excercise: Jackknife estimator for the variance estimators of variance Let x1 , . . . , xn ∼ N (µ, σ 2 ), the two usual estimators of σ 2 : n S2 = 1X (xi − x̄)2 n n and S̃ 2 = i=1 1 X (xi − x̄)2 n−1 i=1 their expectations: E[S 2 ] = n−1 2 σ n Vivien Rossi and E[S̃ 2 ] = σ 2 resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion excercise: Jackknife estimator for the variance estimators of variance Let x1 , . . . , xn ∼ N (µ, σ 2 ), the two usual estimators of σ 2 : n S2 = 1X (xi − x̄)2 n n and S̃ 2 = i=1 1 X (xi − x̄)2 n−1 i=1 their expectations: E[S 2 ] = n−1 2 σ n and E[S̃ 2 ] = σ 2 exercise with R simulate a sample compute the jackknife estimator of S 2 using the function jackknife from package bootstrap compare with the estimator S̃ 2 Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife The bootstrap Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap The bootstrap goal estimate confidence interval of a parameter with or with assumption on data distribution estimate bias of an estimator principle of the nonparametric bootstrap starting point: x1 , . . . , xn i.i.d ∼ L(θ) repeat for k = 1, . . . , K 1 2 sample with replacement x1∗ , . . . , xn∗ among x1 , . . . , xn compute θk∗ the estimates of θ with x1∗ , . . . , xn∗ θ1∗ , . . . , θK∗ inference on θ many variants parametric bootstrap smooth bootstrap . . . Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion The bootstrap: generalities In Jackknife, the number of resamples is confined by the number of observations (n-1). In bootstrap, the original sample could be duplicated as many times as the computing resources allow, and then this expanded sample is treated as a virtual population. Unlike cross validation and Jackknife, the bootstrap employs sampling with replacement In cross-validation and Jackknife, the n in the sub-sample is smaller than that in the original sample In bootstrap every resample has the same number of observations as the original sample. Thus, the bootstrap method has the advantage of modeling the impacts of the actual sample size Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap example: median estimation based on bootstrap sample: 10, 15, 17, 8 and 11 → median estimate = 11 resampling 8 11 15 10 8 10 15 10 10 11 15 17 10 10 17 11 15 10 10 17 17 17 15 17 10 10 15 10 17 10 10 8 8 8 15 median 10 10 15 11 15 10 8 median estimate by bootstrap: mean(10,10,15,11,15,10,8)=11.29 sd of median estimate by bootstrap: sd(10,10,15,11,15,10,8)=2.69 Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: median estimation based on bootstrap sample: 10, 15, 17, 8 and 11 → median estimate = 11 resampling 8 11 15 10 8 10 15 10 10 11 15 17 10 10 17 11 15 10 10 17 17 17 15 17 10 10 15 10 17 10 10 8 8 8 15 median 10 10 15 11 15 10 8 median estimate by bootstrap: mean(10,10,15,11,15,10,8)=11.29 sd of median estimate by bootstrap: sd(10,10,15,11,15,10,8)=2.69 exercise with R estimate the confidence interval using the function bootstrap from package bootstrap Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: parametric bootstrap principle of the parametric bootstrap starting point: x1 , . . . , xn i.i.d ∼ L(θ) where L known compute θ̂ the mle of θ repeat for k = 1, . . . , K 1 2 sample x1∗ , . . . , xn∗ i.i.d ∼ L(θ̂) compute θk∗ the mle of θ with x1∗ , . . . , xn∗ θ1∗ , . . . , θK∗ inference on θ Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: parametric bootstrap principle of the parametric bootstrap starting point: x1 , . . . , xn i.i.d ∼ L(θ) where L known compute θ̂ the mle of θ repeat for k = 1, . . . , K 1 2 sample x1∗ , . . . , xn∗ i.i.d ∼ L(θ̂) compute θk∗ the mle of θ with x1∗ , . . . , xn∗ θ1∗ , . . . , θK∗ inference on θ Exercise with R: the exponential distribution simulate a sample from Exp(λ) with λ = 5 estimate λ using both parametric bootstrap and nonparametric bootstrap (function boot from package boot) Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap example: resampling residual principle of the resampling residual starting point: (x1 , y1 ), . . . , (xn , yn ) and a model f (·, θ) calibrate the model → θ̂: ŷi = f (xi , θ̂) repeat for k = 1, . . . , K 1 2 εi = ŷi − yi ∀i, yi∗ = yi + ε̃ with ε sampled from ε1 , . . . , εn calibrate the model with (x1 , y1∗ ), . . . , (xn , yn∗ ) θ1∗ , . . . , θK∗ θk∗ inference on θ exercise with R: the case of the linear model simulate data according to a linear model estimate regression parameters using the function lm.boot from the package simpleboot Vivien Rossi resampling Conclusion overview Randomisation tests Cross-validation Jackknife 1 Principle and mecanism of the resampling methods 2 Randomisation tests 3 Cross-validation 4 Jackknife 5 Bootstrap 6 Conclusion Vivien Rossi resampling Bootstrap Conclusion overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion Rationale of supporting resampling Empirical empirical-based resampling do not require assumptions on the sample or the population. Clarity resampling is clean and simple. High mathematical background is not required to comprehend it Small sample size Distributional assumptions requierd by classical procedures are usually met by a large sample size. Bootstrapping could treat a small sample as the virtual population to "generate" more observations Non-random sample Resampling is valid for any kind of data, including random and non-random data. Large sample size Given a very large sample size, one can reject virtually any null hypothesis → divide the sample into subsets, and then apply a simple or double cross-validation. Replications Repeated experiments in resampling such as cross-validation and bootstrap can be used as internal replications. Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion Criticisms of resampling Assumption "You’re trying to get something for nothing". Every theory and procedure is built on certain assumptions and requires a leap of faith to some degree. Indeed, the classical statistics framework requires more assumptions than resampling does Generalization resampling is based on one sample and therefore the generalization cannot go beyond that particular sample. Bias confidence intervals obtained by simple bootstrapping are always biased though the bias decreases with sample size. (for normal case the bias in is at least n/(n-1)) Accuracy for small sample resampling may be less accurate than conventional parametric methods. Not very convincing argument because today computers are very powerful. pros and cons in both traditional and resampling methods carry certain valid points. → the appropriateness of the methodology highly depends on the situation Vivien Rossi resampling overview Randomisation tests Cross-validation Jackknife Bootstrap Conclusion R packages for resampling methods boot quite a wide variety of bootstrapping tricks. bootstrap relatively simple functions for bootstrapping and related techniques. coin permutation tests Design includes bootcov for bootstrapping the covariance of estimated regression parameters and validate for testing the quality of fitted models by cross validation or bootstrapping. MChtest Monte Carlo hypothesis tests: tests using some form of resampling. meboot a method of bootstrapping a time series. permtest a function for permutation tests of microarray data. resper for doing restricted permutations. scaleboot produces approximately unbiased hypothesis tests via bootstrapping. simpleboot performs bootstraps in simple situations: one and two samples, and linear regression. Vivien Rossi resampling
![arXiv:1501.06623v1 [q-bio.PE] 26 Jan 2015](http://s1.studyres.com/store/data/003660370_1-c3fe9f4f5d3b3a85fe075a428636185e-150x150.png)
