Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download 2. Hypothesis testing and inference 0.5cm
Transcript
Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... 2. Hypothesis testing and inference Important concepts in statistics Kim-Anh Lê Cao The University of Queensland Diamantina Institute Brisbane, Australia .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Summary Last episode’s highlights I ▶ Descriptive statistics are as important as inferential statistics ▶ Different types of measurement influence the methods for summarizing and displaying the information (also influences the statistical test to use, see next courses) ▶ Shape and characteristics of distributions are important: skewed or bimodal distributions require different descriptive stats Measures of central tendency and variability ▶ Difference between population and sample parameters SD and SEM are very different measures .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Summary Last episode’s highlights II ▶ Correlation and covariance measures Pearson ̸= Spearman’s correlation Correlation does not imply causation Correlation and variance-covariance matrices Variance-covariance matrix is an unstandardized version of the correlation matrix ▶ Graphics Q-Q plot Boxplot What is the appropriate graphical output that will give insight into my data? .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Table of Contents 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Population vs. sample Population vs. sample In Statistics, we rely on a sample (small subset of a larger set of data) to draw inferences about the larger set (the population). ▶ define the population to which inferences are sought ▶ design a study in which the biological samples have been appropriately selected .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Population vs. sample Population vs. sample Definition A population is a complete set of individuals or objects that we want information about. Definition Since this is too expensive, or impossible, we rely on a sample, a subset of the population, and use the data from the sample to infer something about the population as a whole. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Sampling Definition We take a random sample from the population so that the sample is representative of the population of interest and is also unbiased. Examples of types of sampling: simple random sampling, systematic sampling, stratified sampling, cluster sampling, ... Definition Selection bias occurs when the sample itself is unrepresentative of the population we are trying to describe. Example: selecting individuals with certain characteristics and excluding any that deviate from these elements. MMR vaccine: efficacy of the vaccine on 12 children with behavioural disorders, but no attempt made to look at children without behavioural disorders to see if their exposure to the MMR vaccine was . . . greater . . . . . or . . .less. . . . .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Random variable Definition A random variable (also called stochastic variable) is a random process with a numerical outcome, i.e. a variable whose value is subject to variations due to chance. ▶ A random variable does not have a single fixed value, but a set of possible different values. Each of them are associated to a probability. Definition A discrete random variable is a random variable with discrete outcome, i.e. it assumes any of a specified list of exact values. A continuous random variable assumes any numerical value in an interval or collection of intervals (continuous outcome). rf. Course #1: types of variables .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Random variable Definition The values of a random variable can be summarised in a frequency distribution called a probability distribution. 0.2 0.0 0.1 normal density 0.3 0.4 (a) µ Figure : A probability distribution assigns a probability to each measurable subset of the possible outcomes of a random experiment. rf. Course #1: density plot of a variable .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Table of Contents 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The normal distribution The normal distribution The Normal distribution models continuous random variables that commonly occur. ▶ Built on the central limit theorem (See next). ▶ Symmetrical relative to the mean, mode, and median (all equal to µ) We denote by X ∼ N (µ, σ 2 ) a random variable X that follows a Normal distribution with mean µ and variance σ 2 Example: X = height in inches of 12-years old female students in a school in Brisbane, X ∼ N (59, 52 ) .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The normal distribution The normal distribution I Figure : There exists an infinity of Normal distributions with different means and different standard deviations. Example of density plots for four normally distributed random variables (source: www.wikipedia.org). ▶ Many measurements are normally distributed or close to it: height, length, growth, measurement errors, ... .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The normal distribution The normal distribution II ▶ The normal distribution is easy to work with mathematically. In many practical cases, the methods developed using normal theory work quite well even when the distribution is not normal. ▶ Many sampling distributions based on large n can be approximated by the normal distribution even though the population distribution itself is definitely not normal (see CLT). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The standard normal distribution The standard normal distribution There is an infinite number of Normal distributions (with different means and different SD)! → Let’s standardize to a N (0, 1). Definition Definition of the Z-score. If X ∼ N (µ, σ 2 ), then Z= X −µ σ so that Z ∼ N (0, 1). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The standard normal distribution The standard normal distribution We have Z= X −µ σ Interpretation of the Z values: ▶ If Z = 2, the corresponding X value is exactly 2 standard deviations above the mean. Rearranging the terms gives: X = Z ∗ σ + µ = 2σ + µ .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The standard normal distribution The standard normal distribution We have Z= X −µ σ Interpretation of the Z values: ▶ If Z = 2, the corresponding X value is exactly 2 standard deviations above the mean. Rearranging the terms gives: X = Z ∗ σ + µ = 2σ + µ ▶ If Z = 0, X = the mean, i.e. µ. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The standard normal distribution The standard normal statistical table Appendix II reports the cumulative normal probabilities for variable with a normal N (0, 1) distribution (i.e. Z-scores). ▶ Our random variable Z ∼ N (0, 1) ▶ We want to know the probability P(Z ≤ z), where z can take any value we want ▶ P(Z ≤ z) is the area under the curve highlighted in blue i.e. proportion of the distribution below the value z ▶ .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The standard normal distribution The standard normal statistical table Normal cumulative distribution function: P(Z ≤ z) when Z ∼ N (0, 1) 2nd decimal place of z 1rst decimal of z 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.0 0.5000 0.5040 0.5080 0.5120 0.5160 0.5199 0.5239 0.5279 0.5319 0.1 0.5398 0.5438 0.5478 0.5517 0.5557 0.5596 0.5636 0.5675 0.5714 0.2 0.5793 0.5832 0.5871 0.5910 0.5948 0.5987 0.6026 0.6064 0.6103 0.3 0.6179 0.6217 0.6255 0.6293 0.6331 0.6368 0.6406 0.6443 0.6480 0.4 0.6554 0.6591 0.6628 0.6664 0.6700 0.6736 0.6772 0.6808 0.6844 0.09 0.5359 0.5753 0.6141 0.6517 0.6879 Example For a probability = 0.5, half the area of the standardized normal curve lies to the left of z = 0. For z = 0.46, we have P(Z ≤ 0.46) = 0.6772 Only positive values of z are reported, since the the normal distribution is symmetric, e.g. P(Z ≥ 0.46) = 1 − P(Z ≤ 0.46) = 1 − 0.6772 = 0.3228 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . NOTE: While memorization may be useful, you will be much better off if you gain an Definitions asCommon TheTry Central Theorem Hypothesis testing Multiple testing Summary itive understanding to why thedistributions rules that follow are correct. drawingLimit pictures of the . . . . . . to convince . . . . yourself . . . . .that . . each . . . rule . . .is. valid. ... ........ ................... .......... ... mal. distribution The standard normal distribution LES: The standard normal statistical table: some properties P(Z a) = F(a) (use when a is positive) P(Z 1.0) = F(1.0) = .84 = 1 - F(-a) (use when a is negative) P(Z -1.0) = F(-1.0) = 1 - F(1.0) = .16 u can also easily work in the other direction, and determine what a is given P(Z Find a for P(Z a) = .6026, .9750, .3446 a) 1 To solve: for p .5, find the probability value in Table I, and report the corresponding ue for Z. For p < .5, compute 1 - p, find the corresponding Z value, and report the negative of value, i.e. -Z. P(Z .26) = .6026 P(Z 1.96) = .9750 P(Z -.40) = .3446 (since 1 - .3446 = .6554 = F(.40)) 2 P(Z ≤ z) = 1 − P(Z ≤ −z) 3 P(Z ≥ z) = 1 − P(Z ≤ z) TE: It may be useful to keep in mind that F(a) + F(-a) = 1. P(Z a) = 1 - F(a) = F(-a) Find P(Z (use when a is positive) (use when a is negative) a) for a = 1.65, -1.65, 1.0, -1.0 To solve: for positive values of a, look up and report the value for F(a) given in Appendix able I. For negative values of a, look up the value for F(-a) (i.e. F(absolute value of a)) and ort 1 - F(-a). P(Z P(Z P(Z ≤ z) 1.65) = F(1.65) = .95 -1.65) = F(-1.65) = 1 - F(1.65) = .05 Find P(Z a) for a = 1.5, -1.5 https://www3.nd.edu/˜rwilliam/stats1/ To solve: for a positive, look up F(a), as before, and subtract F(a) from 1. distribution For a negative, Normal - Page 3 report F(-a). Kim-Anh Lê Cao P(Z 1.5) = 1 - F(1.5) = 1 - .9332 = .0668 2015-1.5) Statistics frightened bioresearchers #2 P(Z = F(1.5)for = .9332 .. . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 Exercise We recorded the mean 24h systolic pressure in mm Hg in healthy individuals. We assume the random variable X = ’mean 24h systolic pressure’ is normally distributed, with µ = 120 and σ = 10, i.e. X ∼ N (µ, σ 2 ) 1 1 Describe the appropriate transformation of the data 2 Sketch the distributions to illustrate the answers (no exact answer is required) 1 . . . . . . . . . . . . . . . .. .. .. .. .. .. .. In this exercise, we assume these parameters to be the known .. .. .. .. .. .. .. .. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 - Q1 1 What area of the curve is above 130 mm Hg? .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 - Q1 1 What area of the curve is above 130 mm Hg? ▶ We have X ∼ N (µ = 120, σ 2 = 102 ). We want: P(X ≥ 130). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 - Q1 1 What area of the curve is above 130 mm Hg? ▶ We have X ∼ N (µ = 120, σ 2 = 102 ). We want: P(X ≥ 130). We calculate the corresponding critical standardised z-value using the z-score: z = 130−µ = 130−120 =1 σ 10 ▶ .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 - Q1 1 What area of the curve is above 130 mm Hg? ▶ We have X ∼ N (µ = 120, σ 2 = 102 ). We want: P(X ≥ 130). We calculate the corresponding critical standardised z-value using the z-score: z = 130−µ = 130−120 =1 σ 10 Represent the critical value on a N (0, 1) density plot: ▶ ▶ 0.2 0.3 0.4 Q1 0.0 0.1 area = 0.159 −4 −2 0 2 4 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 - Q1 1 What area of the curve is above 130 mm Hg? ▶ We have X ∼ N (µ = 120, σ 2 = 102 ). We want: P(X ≥ 130). We calculate the corresponding critical standardised z-value using the z-score: z = 130−µ = 130−120 =1 σ 10 Represent the critical value on a N (0, 1) density plot: ▶ ▶ 0.2 0.3 0.4 Q1 0.0 0.1 area = 0.159 −4 ▶ −2 0 2 4 Using a software or a statistical table, and using property 3, P(Z ≥ 1) = 1 − P(Z ≤ 1) = 1 − 0.8413 = 0.159 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 8 - Q2 2 What area of the curve is between 100 and 140 mm Hg? 3. P(a EX: Find P(a Z b)4: for P(a a = -1 ≤ and Z b =≤ 1.5 b) Property ▶ Z b) = F(b) - F(a) = P(Z ≤ b) − P(Z ≤ a) To solve: determine F(b) and F(a), and subtract. P(-1 Z 1.5) = F(1.5) - F(-1) = F(1.5) - (1 - F(1)) = .9332 - 1 + .8413 = .7745 Calculate the critical standardised values a and b 4. Represent For a positive, Z a) standardised = 2F(a) - 1 ▶ theP(-a critical values on a density plot PROOF: distribution of a N (0, 1) P(-a Z a) ▶ Use a software or the(bystatistical table of a N (0, 1) to obtain = F(a) - F(-a) rule 3) = F(a) (1 - F(a)) P(Z ≤- b) and P(Z ≤(bya)rule 1) = F(a) - 1 + F(a) = 2F(a) - 1 Kim-Anh Lê Cao 2015 Statistics frightened #2 a = 2.58 EX:for find P(-a Zbioresearchers a) for a = 1.96, .. . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. See Lecture notes . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... The Empirical Rule The Empirical Rule ▶ ▶ About 2/3 of all cases fall within one standard deviation of the mean: P(µ − σ ≤ X ≤ µ + σ) = 0.6826 About 95% of cases lie within 2 standard deviations of the mean: P(µ − 2σ ≤ X ≤ µ + 2σ) = 0.9544 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Other continuous distributions Uniform distribution The Uniform distribution is a continuous probability distribution in which all values within a specified interval have the same probability. Used to model random events when each potential outcome or occurrence has equal probability of occurring. Distribution of species: penguins exhibit uniform spacing by defending their territory among their neighbors. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Other continuous distributions The t-distribution ▶ similar to a normal distribution when large number of samples ▶ used in t-tests see Course # 4 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Other continuous distributions The F-distribution ▶ used in ANOVA and F-tests see Course # 5 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Other continuous distributions The Chi-squared distribution ▶ distribution of a sum of the squares of k independent standard normal random variables ▶ used in goodness of fit of an observed distribution to a theoretical one ▶ used when testing the independence of two qualitative variables. see Course # 4 and 5 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Discrete distributions Poisson distribution The Poisson distribution is a discrete probability distribution which models the probability of the number of events occurring over a fixed interval of time or space given a known mean. ▶ # of accidents at an intersection ▶ # of birth defects ▶ # of kangaroos in a square kilometer ▶ models rare occurrences ▶ called the ‘law of small numbers’: the event does not happen often, but there are many opportunities for it to occur. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Discrete distributions Binomial distribution The Binomial distribution is a discrete probability distribution modelling the number of successes in a sequence of several n independent experiments, each of which have the same probability p of success. ▶ ▶ # of people in a clinical study who died of heart disease ▶ # of animals in a population that carry a certain genetic trait describes occurrences, not magnitude. It may model how many participants finished a race, not how fast the participants were. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 .. . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Discrete distributions Summary: discrete probability distributions Poisson distribution: ▶ assumes that the mean and variance are the same. Overdispersion: data show variance > the mean Negative binomial distribution: ▶ overdispersion ▶ two parameters to estimate → The Poisson distribution is a special case of the negative binomial distribution. Binomial distribution: ▶ approximate a Poisson distribution for a large n and a small probability p of success. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Discrete distributions Which distribution? .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Discrete distributions To add more confusion: link between distributions .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Table of Contents 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Theorem and implications Central Limit Theorem Theorem Given a population with mean µ and standard deviation σ, the sampling distribution of the mean based on repeated random samples of size n from that same population has the same mean µ √ (the mean of the means) and a standard deviation of σ/ n (the standard error of the mean). i.e, If X = (X1 , X2 , . . . Xn ) is a sequence of i.i.d.∗ random variables with each Xi following a given distribution with mean µ and standard deviation σ then X̄ ∼ N (µ, σ 2 /n) Consequence: no matter what type of distribution the population follows, the sampling distribution of the mean from that same population will be approximately (or exactly) normally distributed. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 ∗ .. . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. independently and identically distributed . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Theorem and implications How large should the sample size be for the Central Limit Theorem to hold? A sample size > 30 (depends on the population distribution, symmetric: n < 30, heavily skewed: n > 30). ▶ In practice, selecting repeated samples of size n and generating a sampling distribution for the mean is not necessary. ▶ Only one sample is selected, and we calculate the sample mean as an estimate of the sample population. ▶ We can invoke the CLT if the sample size is > 30 to justify that the sampling distribution of the mean is known (See Exercise 9). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Examples Example 9 (a) The population is assumed to be normal with mean 0 and standard deviation 1 (N (0, 1)) therefore, the true population mean µ is equal to 0. 2.5 (a) 1.5 1.0 0.0 0.5 Density 2.0 population n=5 n = 10 n = 20 n = 30 −4 −2 0 2 4 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Examples Example 9 (b) The population is assumed to follow a χ2 distribution with 2 degrees of freedom, therefore the true population mean is equal to 2 (theoretical result from a χ22 distribution). 1.5 (b) 0.0 0.5 Density 1.0 population n=5 n = 10 n = 20 n = 30 −2 0 2 4 6 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Examples Example 9 (c) The population is assumed to follow a uniform distribution, therefore, the true population mean is equal to 0.5 (theoretical result from a uniform distribution). 8 (c) 4 0 2 Density 6 population n=5 n = 10 n = 20 n = 30 −0.5 0.0 0.5 1.0 1.5 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... SD vs. SEM SD vs. SEM population n=5 n = 30 σ/√n" σ" μ" Figure : Distribution of the mean from a normally distributed population (in blue). The true population mean is represented as a vertical line. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... SD vs. SEM SD vs. SEM population n=5 n = 30 σ/√n" σ" μ" The CLT illustrates the difference between SD and SEM. ▶ The SD is related to the original population (or one sample thereof). ▶ The SEM is related to the mean of repeated samplings from the original population. ▶ When the number of samples increases, the SEM decreases as the distribution of the means converge to the real mean population µ. See course #1 for a definition of SD and SEM .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Exercise Exercise 9 Exercise The average GPA at a particular school is 2.89 with a standard deviation of 0.63. If we take a random sample of 35 students, what is the probability that the average GPA for this sample is greater than 3.0? ▶ ▶ ▶ ▶ Define the random variable X = ‘GPA at a particular school’ and determine its distribution Use the CLT which states that X̄ follows a normal distribution with a specific mean and standard deviation Use a z-score transformation for the mean Use a software or statistical table to calculate the probability See lecture notes for the answers .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Table of Contents 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Hypothesis testing is an essential part of statistical inference ▶ Two competing hypotheses: the null hypothesis, H0 , against the alternative (or research) hypothesis, H1 . ▶ We carry out an experiment to reject the null hypothesis: H0 : there is no difference in taste between black tea and green tea, against H1 : there is a difference in taste between black tea and green tea. ▶ Hypotheses are often statements about population parameters∗ , or a statement about the distribution of variable of interest. ∗ See upcoming Courses #5 and #6 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Steps in hypothesis testing Definition A statistical test is based on the concept of proof by contradiction and is composed of the five steps: 1 State the null hypothesis H0 , 2 State the research hypothesis (alternative hypothesis): H1 , 3 Compute the test statistic (T.S) related to H0 , 4 Define the rejection region, 5 Check assumptions and draw conclusions. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing The null and alternative hypothesis ▶ H0 represents a theory is believed to be true (or has not been proved yet). ▶ H1 states the statistical hypothesis test to establish. Example In a clinical trial of a new drug, the null hypothesis might be that the new drug is no better, on average, than the current drug: H0 : there is no difference between the two drugs on average against H1 : the two drugs have different effects, on average (two-sided test), or, H1 : the new drug is better than the current drug, on average (one-sided test). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Test statistics The null hypothesis is statistically tested against the alternative using a suitable distribution of a statistic. ▶ The statistic is computed from the experimental data. ▶ We compare the statistic with its theoretical distribution and draw a conclusion with respect to the null hypothesis → reject H0 or do not reject H0 ▶ The choice of a test statistic depends on the assumed probability model and the hypotheses tested. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Type I and type II errors Definition The probability of wrongly rejecting H0 when in fact H0 is true is the significance level, generally denoted α and usually set to α = 5% .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Type I and type II errors ▶ ▶ ▶ A type I error is committed: reject the null hypothesis when it is true. Probability of a type I error is α. A type II error is committed if we do not reject the null hypothesis when it is false and the research hypothesis is true. Probability of a type II error is β. The power of a statistical test is the probability that it correctly rejects the null hypothesis when the null hypothesis is false (i.e. the ability of a test to detect an effect if the effect actually exists). The power is equal to 1 − β. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Critical value Definition The critical value(s) for a hypothesis test is a threshold to which the value of the test statistic in a sample is compared to a theoretical value to decide whether or not the null hypothesis is rejected. The critical value for any hypothesis test depends on the significance level at which the test is carried out, and whether the test is one-sided or two-sided (here indicated by the z value). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Critical region Definition The critical region or rejection region, is a set of values of the test statistic for which the null hypothesis is rejected. If the observed value of the test statistic belongs to the critical region, we conclude ‘Reject H0 ’ otherwise we ‘Do not reject H0 ’. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Steps in hypothesis testing Conclusion The final conclusion once the test has been carried out is always given in terms of the null hypothesis. We either ”Reject H0 in favour of H1 ” or we ‘Do not reject H0 ’. We never conclude ‘Reject H1 ’, or even ‘Accept H1 ’. Remark If we conclude ‘Do not reject H0 ’, this does not necessarily mean that the null hypothesis is true, it only suggests that there is not sufficient evidence against H0 in favour of H1 . Rejecting the null hypothesis then, suggests that the alternative hypothesis may be true. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Example Example 10 Illustrative example using a z-test and the CLT, where we assume that the standard deviation is known. Example An agricultural service wants to determine whether the mean yield per acre for a particular variety of soybeans has increased since last year. Last year, across all farms, the mean yield was 520 bushels/acre with a standard deviation s = 124. This year we have a sample of n = 36 one-acre plots, from which the sample mean yield is x̄ = 573. Can we conclude that the mean yield for all farms is above the target from last year of 520? → Given: X is the yield/acre and X ∼ N (520, 1242 ) → CLT theorem: X̄ ∼ N (520, 1242 /36) .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Example Example 10 1 2 3 4 H0 : µ ≤ 520 H1 : µ > 520 x̄−520 √ , with z ∼ N (0, 1) (CLT theorem). T.S: z = 124/ 36 Rejection region: f (x) area α µ = 520 acceptance region rejection region .. Kim-Anh Lê Cao . .. we reject H0 if z ≥ zα , with α = 0.05. 2015 Statistics for frightened bioresearchers #2 . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Example f (x) area α µ = 520 acceptance region rejection region Rejection region: we reject H0 if z ≥ zα , with α = 5%. ▶ Determine the location of the rejection area: → determine the z value that has an area α to its right ▶ A statistics table gives the critical value of the theoretical test statistics under H0 T .Sα = 1.645 ▶ From our data we have T .S = z = x̄−µ √ σ/ n = 573−520 √ 124/ 36 = 2.56 5 Since we have z ≥ zα , we reject H0 in favor of the alternative hypothesis. Conclusion: the average soybean yield per acre is significantly greater than last year’s target 520. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... One or two-sided test One-sided test (a) f (x) µ = 520 rejection region In Example 10, we set up a one-side test. Instead of H1 : µ ≥ 520, we could have set H1 : µ ≤ 520, ▶ small values of x̄ would indicate the rejection of null hypothesis, ▶ the rejection region would be located in the lower tail of the distribution of x̄ Kim-Anh Lê Cao → we reject H0 if z ≤ zα . .. 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... One or two-sided test Two-sided test (b) f (x) µ = 520 rejection region rejection region A two-sided test could be formulated for H1 : µ ̸= 520: ▶ both large and small values of x̄ contradict the null hypothesis, ▶ the rejection region is located in both tails of the distribution of x̄ → we reject H0 if z ≤ zα/2 and z ≥ zα/2 equiv. to: we reject H0 if |z| ≥ zα/2 . .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... One or two-sided test One or two-sided test Remark The one-sided test has the ‘advantage’ over the two-tailed test of obtaining statistical significance with a smaller departure from the hypothesized value, as there is interest in only one direction. One-tailed tests are appropriate when it is not important to distinguish between no effect and an effect in the unexpected direction. The use of a one-sided test must be done with caution as its use can be controversial. In medical research, we frequently choose a two-sided test, even if we have an expectation about the direction of the test. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... One or two-sided test Example of one-sided test A researcher is only be interested in whether a treatment is better than a placebo control. → it would not be worth distinguishing between the cases: ▶ in which the treatment was worse than a placebo, or ▶ in which it was the same because in both cases the drug would be worthless. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... p-value p-value Rejection region and p-value are equivalent. The p-value determines the significance of the results and is obtained with a statistical software. Definition The p-value is the probability of obtaining a test statistic result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. The p-value is a number between 0 and 1: ▶ If p-value ≤ α, we reject H0 : there is strong evidence against H0 . ▶ If p-value > α, we fail to reject H0 : there is weak evidence against H0 . Note that usually the significance level α is set to 5%. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 .. . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... p-value Type I, type II errors and power calculation I ▶ Type I error α: reject the null hypothesis when it is true. Probability of a type I error is α. ▶ Type II error β: we do not reject the null hypothesis when it is false and the research hypothesis is true. ▶ Power 1 − β of a statistical test: probability of a test to detect an effect if the effect actually exists (i.e. the ability of a test to detect an effect if the effect actually exists). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... p-value Type I, type II errors and power calculation II G*Power software: ▶ Choose the type of statistical test (I do not recommend you use that software until we finish that course series!) ▶ One or two tailed test ▶ Input α and power 1 − β ∗ .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . . .upcoming . . . . . . . Course . . . . . #5 . See .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Table of Contents 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Multiple testing matters Multiple testing occurs in large data sets Suppose we perform 10,000 separate hypothesis tests (one for each gene). If we use a standard p-value cut-off of 5%, then we expect 500 genes to be deemed “significant” by chance. If we perform m hypothesis tests, what is the probability of at least 1 false positive? ▶ P(making an error) = α (α is the significance level) ▶ P(not making an error) = 1 - α ▶ P(not making an error in m tests) = (1 − α)m ▶ P(making at least 1 error in m tests) = 1 − (1 − α)m If m = 5, P(at least 1 error in 5 tests) = 0.22. If m = 10, P(at least 1 error in 10 tests) = 0.40. → this is not the α = 5% as we originally requested! .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Multiple testing matters ▶ ▶ ▶ The probability of falsely rejecting at least one of the hypotheses increases as the number of tests increases. Even if probability of type I error is α = 5% for each individual test, the probability of falsely rejecting at least one of those tests is larger than 5%. Applies for high throughput experiments but also for a modest number of comparisons (e.g. Tukey’s tests∗ ) ∗ See Course #6 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Multiple testing matters ‘Adjusting for multiple testing’ Means ‘controlling the Type I error rate’. Different methods proposed which differ in fundamental ways: FWER, FDR, etc∗ . Table : Number of errors committed when performing m hypothesis tests. m0 is the number of true null hypothesese (‘do not reject H0 ’) and R is the number of rejected null hypotheses, V is the number of false positives. Not called significant Called significant Null true U V m0 Alternative True T S m − m0 Total m−R R m Methods often try to correct or estimate V or R. ∗ See Section 5 in lecture notes for a detailed list .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... FWER FWER I Family-wise error rate is the probability of at least one type I error: FEWR = P(V ≥ 1), e.g. Bonferroni correction: ▶ ▶ ensures that the overall type I error rate α is maintained rejects any hypothesis with a p-value ≤ α/m. Example If m = 10, 000, then we need a p-value 0.05/10000 = 5 ∗ 10−06 to declare significance. The adjusted Bonferroni p−value is p̃j = min(m ∗ pj , 1), pj is the raw p−value of the jth test over m tests and p̃j is the Bonferroni adjusted p-value. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 .. . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... FWER FWER II ▶ Bonferroni adjustment is very stringent. ▶ Counter-intuitive as the interpretation of findings depends on the number of other tests performed. ▶ The general null hypothesis is that all the null hypotheses are true and it has a high probability of Type II errors (i.e. not rejecting H0 when there is a true effect). .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... FDR FDR I The False Discovery Rate (FDR, Benjamini and Hochberg, 1995) controls the proportion of false positives among the set of rejected hypotheses (R). To control for a FDR at level signif level δ: 1 Order the unadjusted p-values: p(1) ≤ p(2) ≤ · · · ≤ p(m) 2 Find the test with the highest rank j for which the p-value p(j) ≤ mj ∗ δ 3 Declare the tests with rank 1, 2, . . . , j as significant. Rank 1 2 3 4 5 6 7 8 9 10 .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. raw p-value 0.0008 0.009 0.165 0.205 0.396 0.450 0.641 0.781 0.900 0.993 . .. . j m ∗ 0.05 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050 . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. Reject H0 yes yes no no no no no no no no . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... FDR FDR II The adjusted FDR Benjamini and Hochberg p−values are defined as: p̃(1) = min(p̃(2) , mp(1) ) .. . m p̃(m−1) = min(p̃(m) , p ) m − 1 (m−1) p̃(m) = p(m) p(j) is the jth ranked raw p−value amongst the m tests (p-values) and p̃(j) is the 1 2 3 4 5 6 7 8 9 10 raw p-values 0.0008 0.0090 0.1650 0.2050 0.3960 0.4500 0.6410 0.7810 0.9000 0.9930 adjusted p-values 0.0080 0.0450 0.5125 0.5125 0.7500 0.7500 0.9157 0.9763 0.9930 0.9930 adjusted p−value ranked jth. (start from the bottom) .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... q-value q-value I Definition The q-value (Storey 2002) is the minimum FDR attained at or above a certain score, i.e. the q-values take only into account the tests with q-values less than the chosen threshold. The q-value complements the FDR. Example In a microarray study testing for differential expression, if a gene X has a q−value = 0.013, it means that 1.3% of the genes with a p-value ≤ 0.013 (i.e. ranked before gene X with decreasing significance) are false positives. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... q-value q-value II Histogram of p-values when performing m tests: (a)$ (b)$ (c)$ (a) We expect to see no significant changes in the experiment. (b) We expect to see significant changes in the experiment. (c) The q-value approach tries to find the threshold where the p-value distribution flattens out and incorporates this threshold into the calculation of the FDR adjusted p-values. → The q-value helps to establish just how many of the significant values are actually false positives .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... q-value Adjusting or not? The goal of multiple comparisons corrections is to reduce the number of false positives. Consequence: we increase the number of false negatives (there really is an effect but it is not detected as statistically significant). → If false negatives are very costly, you may not want to correct for multiple comparisons. Remark If an adjusted p-value is not significant, you can cautiously report as a ‘possible effect of gene X on cancer’. The cost of a false positive – if further experiments are conducted, is a few more experiments. The cost of a false negative, on the other hand, can be to miss an important discovery. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Table of Contents 1 Definitions 2 Common distributions 3 The Central Limit Theorem 4 Hypothesis testing 5 Multiple testing 6 Summary .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Summary Summary ▶ Important definitions of population vs. sample, random variables and probability distribution. ▶ Normal, standard normal distribution and z-score. ▶ The central limit theorem tells us that means of observations, regardless of how they are distributed, begin to follow a normal distribution as the sample size increases. → differences between SD and SEM ▶ Different steps in hypothesis testing, difference between one and two-sided tests, critical region and p-value. ▶ Multiple testing and different approaches to adjust for multiple testing, such as the FWER, FDR and the q-value. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Acknowledgements Acknowledgements ▶ Former Master’s students and workshop trainees ▶ Extra 10 Exercises: Benoı̂t Gautier (UQDI) ▶ Coordination of the short course series: Dr Nick Hamilton (IMB) and Ms Jessica Schwaber (AIBN) ▶ Communication & website: Kate Templeman (UQDI), Gemma Ward (IMB) .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. . Definitions Common distributions The Central Limit Theorem Hypothesis testing Multiple testing Summary .. ..... ..................... ........ ................... .......... ... Questions More Questions? Next course July 13 (AIBN) is a gentle introduction to experimental designs. .. Kim-Anh Lê Cao 2015 Statistics for frightened bioresearchers #2 . .. . .. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. . .. . .. .
