Download Analytical Tools for High Throughput Sequencing Data

Marshall University School of Medicine
Department of Biochemistry and Microbiology
BMS 617
Lecture 15: Sample size and Power
Marshall University Genomics Core Facility
Sample Size
• Virtually all the quantities we calculate from
sample data depend on the size of the sample.
• Small sample sizes lead to wide confidence
intervals
– Consequently we can fail to detect important
differences even when they exist
– Such experiments are called underpowered
• When planning and designing experiments, we
need to quantify the amount we can learn from
the sample size we plan to use.
Marshall University School of Medicine
Sample size: what not to do
• It is quite common for investigators to use an ad-hoc approach to
determining the sample size:
– Collect some data and analyze it
– If the results are not statistically significant, collect some more data
and reanalyze
– Repeat until the results are statistically significant.
• There are major problems with this approach:
– The p-values or confidence intervals from this approach cannot be
interpreted
– The problem is that the experiment is stopped when a statistically
significant result is reached, but not stopped otherwise
– It is possible that a statistically significant result would lose
significance when more data were added, but this is never tested
• In theory, any hypothesis can be found to be statistically significant using this
technique
• Though it may take a very large number of samples
Marshall University School of Medicine
Approaches to Sample Size
Calculations
• There are two valid approaches to managing
sample sizes when planning experiments:
• Classical:
– Perform calculations before the study begins to
determine the number of samples to be used.
– Collect data from those samples and analyze
• Adaptive:
– Interim analyses are performed during the course of
the study, and decisions are made to add more
subjects according to the results of theses analyses
Marshall University School of Medicine
Adaptive Experimental Designs
• More sophisticated versions of the adaptive approach can
modify the proportions of subjects allocated to the
treatment and control groups
• Can even sometimes modify the experimental protocol
(e.g. dose)
• The difference between this and the ad-hoc approach is
that all eventualities are planned before the experiment is
started
• Interpretations of statistical significance take account of the
adaptive process
• Details are highly complex and an area of current
methodological research
– Beyond the scope of this course
Marshall University School of Medicine
Simplest Example: Sample size and
confidence interval for a single mean
• To illustrate a sample size calculation, consider the simple problem
of estimating a single mean of a population from a sample
• Remember we calculated the confidence interval for the mean as
(m-w, m+w) where m is the sample mean and w is a margin of error
• w=t*s/√n
– t* is the critical value from the t-distribution
– s is the sample standard deviation
– n is the sample size
• If we have a margin of error we wish to target, it appears easy
enough to rearrange this equation:
– n=(t*s/w)2
– Note that t* depends on n, but this equation can be solved by
software
Marshall University School of Medicine
Sample Size for other confidence
intervals
• For the difference between two means, the best results are
obtained when the two groups have equal size.
• If we assume equal-sized groups, the formula for the number of
subjects in each group becomes
• n=2(t*s/w)2
• Here s is the pooled standard deviation, s2=s12+s22
• We can similarly flip the equations for confidence intervals for
proportions:
– For a single proportion, n≈4p(1-p)/w2
– For two proportions, the number in each group is n≈8p(1-p)/w2
– p is the anticipated proportion (or average of the anticipated
proportions in the second case)
– If p cannot be estimated, p=0.5 gives the "worst case" or largest value
of n
Marshall University School of Medicine
Practical considerations for sample size
calculations
• These formulae for sample size rely either on estimations of the
standard deviation or the actual sample proportion
• Of course, these are not known until the experiment has been
performed
• Which cannot be done until the sample size is determined…
• In practice, the best way to estimate these is with data from a pilot
experiment
– If this is not available, data from similar published work can be used
– But be aware that published data are more likely to have smaller
variance or proportions further from 50:50 than the "average”
experiment
• Note also that the sample sizes calculate the number of samples
that enter the analysis. You may need to increase this value to
account for subjects that drop out of the study, experiments that
fail, or data removed as outliers
Marshall University School of Medicine
Sample Size and Hypothesis Testing
• The most common type of sample size calculation performed is related to
hypothesis testing.
– It answers the question: "How many samples do I need to achieve statistical
significance”
• This depends on four factors:
1.
2.
3.
4.
The level of statistical significance. This is usually 0.05, but a stricter
definition (say 0.01) would require more samples.
The power. Power is the chance of getting a statistically significant result if a
given effect size actually exists. The higher the power required, the more
samples are required.
The effect size. To achieve statistical significance with very small effect sizes
required very large sample sizes. To achieve statistical significance with large
effect sizes requires fewer samples.
The standard deviation (for comparing means) or actual proportion (for
comparing proportions). If the SD is small, the required sample size will be
smaller. For noisy data (large SD), the sample size will be larger. For
proportions, the sample size required is largest when the proportion is 50%.
Marshall University School of Medicine
Example of Sample Size Calculation
Results
• Sample size calculations are usually required in funding applications
• The funding agency wants to know that the experiment has a
reasonable chance of success
• Also wants to know that the sample size is not "overkill”
• i.e. that replicates are not being run unnecessarily
• An example statement from a grant application might look like this:
In order to determine sample size, we used the variance from a previous
unpublished experiment under the same conditions in the same laboratory. The
variance in log2 expression values in this experiment was 0.4468. We chose a
sample size of 11 subjects per group in order to have 80% power of detecting a 1.5
fold change using a significance level of 0.05.
• The values entered into a sample size calculation program were:
– Significance level = 0.05, Power = 0.8, Effect size = log21.5, sd= 0.4468
Marshall University School of Medicine
Sample Size Calculations in Practice
• In practice, the sample size is not merely constrained by the desired
outcome of the experiment
– Also constrained by practical considerations, such as time, funding, laboratory
space, etc.
• Often an investigator will perform a sample size calculation, and find that
the number of required samples is impractical
– Using ideal values for level of statistical significance, standard deviation,
power, and effect size
• It can be informative to run the sample size calculation over a range of
these values
– Provides information as to what can be learned with a practical sample size
– May come to the conclusion that the experiment is impossible with the
number of subjects
– May learn that the experiment can be performed, but with a limited statistical
power or with the ability only to detect large effect sizes
Marshall University School of Medicine
Sample Sizes for Other Analyses
• Sample size calculations even for simple
analyses are complex
• For more complicated analyses they may be
prohibitively complex, or simply not
computable
• For these we resort to some standard rules of
thumb
Marshall University School of Medicine
Sample Sizes for Non-parametric tests
• Sample size calculations can be performed for
non-parametric tests, but only if assumptions
are made about the distribution
• In this case, it is better to use a parametric
test anyway
• As long as the sample sizes are reasonably
large (a few dozen), a good rule of thumb is:
– Compute the sample size for the corresponding
parametric test, and add 15%
Marshall University School of Medicine
Sample Sizes for Multiple Regression
• Some exact calculations are possible for multiple (and
logistic) regression, but these depend on values it is
extremely difficult (or impossible) to estimate
– For example, the degree of correlation between the variables
• In general, you need many more subjects than variables
– At least ten times as many
– Preferably twenty times as many
– And double if you plan to use variable selection
• The number of variables includes all variables used in
variable selection, even if they don't appear in the final
model
• For logistic regression, the number of subjects is the
number falling into the smallest outcome category
Marshall University School of Medicine
Summary
• Sample size calculations are an essential part of
experimental design and planning
• In their simplest form, sample size calculations merely
involve rearranging equations for confidence intervals
• However, they typically require estimates of values
which are not really known until the experiment is
complete
• Computing sample sizes over a range of realistic values
is often informative
• For complex analyses, use standard rules of thumb if
sample size calculations are not available
Marshall University School of Medicine