Download t–test - Bioinformatics.ca

Canadian Bioinformatics
Workshops
www.bioinformatics.ca
Module #: Title of Module
2
Module 6
Hypothesis testing
Exploratory Data Analysis of Biological Data using R
†
Boris Steipe
Toronto, May 22. and 23. 2014
DEPARTMENT OF
BIOCHEMISTRY
DEPARTMENT OF
MOLECULAR GENETICS
†
Oedipus ponders the riddle of the Sphinx. Classical (~400 BCE)
This workshop includes material
originally developed by Raphael Gottardo, FHCRC
and by Sohrab Shah, UBC
Hypothesis testing
Once we have a statistical model that describes
the distribution of our data, we can explore data
points with reference to our model.
In hypothesis testing we typically ask questions
such as:
Is a particular sample a part of the distribution, or
is it an outlier?
Can two sets of samples have been drawn from
the same distribution, or did they come from
different distributions?
Module 6: Hypothesis testing
bioinformatics.ca
Hypothesis testing
Hypothesis testing is confirmatory data analysis, in
contrast to exploratory data analysis.
Concepts:
Null – and Alternative Hypothesis
Region of acceptance / rejection and critical value
Error types
p - value
Significance level
Power of a test (1 - false negative)
Module 6: Hypothesis testing
bioinformatics.ca
Null hypothesis / Alternative hypothesis
The null hypothesis H0 states that nothing of
consequence is apparent in the data distribution.
The data corresponds to our expectation. We
learn nothing new.
The alternative hypothesis H1 states that some
effect is apparent in the data distribution. The data
is different from our expectation. We need to
account for something new. Not in all cases will
this result in a new model, but a new model
always begins with the observation that the old
model is inadequate.
Module 6: Hypothesis testing
bioinformatics.ca
Test types
Just like the large variety of types of hypotheses,
the number of test is large. The proper application
of tests can be confusing and it is easy to make
mistakes.
Common types of tests
A one-sample test compares a sample with a population.
A two-sample test compares samples with each other.
Paired sample tests compare matched pairs of observations with
each other. Typically we ask whether their difference is significant.
...
Module 6: Hypothesis testing
bioinformatics.ca
Test types
... common types of tests
A Z–test compares a sample mean with a normal distribution.
A t–test compares a sample mean with a t- distribution and thus
relaxes the requirements on normality for the sample.
Nonparametric tests can be applied if we have no reasonable model
from which to derive a distribution for the null hypothesis.
Chi–squared tests analyze whether samples are drawn from the same
distribution.
F-tests analyze the variance of populations (ANOVA).
Module 6: Hypothesis testing
bioinformatics.ca
Error types
Truth
Decision
Accept H0
Reject H0
Module 6: Hypothesis testing
H0
1-

"False positive"
"Type I error"
H1

"False negative"
"Type II error"
1-
bioinformatics.ca
introduction
• One sample and two sample t-tests are used
to test a hypothesis about the mean(s) of a
distribution.
• Gene expression: Is the mean expression
level under condition 1 different from the
mean expression level under condition 2?
• Assume that the data are from a normal
distribution.
Module 6: Hypothesis testing
bioinformatics.ca
one sample t-test
t-tests apply to n observations that are independent and normally
distributed with equal variance about a mean .
H0 :m = m
0
H1 : m ¹ m
0
y-m
y-m
t=
»
SE y
s n
0
The 1-sample t-statistic is defined as:
0
i.e. t is the difference in sample mean and 0, divided by the Standard Error
of the Mean, to penalize noisy samples.
If the sample mean is indeed 0, t follows a t-distribution with n-1 degrees of
freedom.
Module 6: Hypothesis testing
bioinformatics.ca
example – HIV data
Is gene 1 differentially expressed?
Is the mean log ratio equal to zero?
# One sample t-test
data <- log(read.table(file="hiv.raw.data.24h.txt",
sep="\t", header=TRUE))
# Compute the log ratios
M <- (data[, 1:4] - data[, 5:8])
gene1 <- t.test(M[1, ], mu=0)
x <- seq(-4, 4, 0.1)
f <- dt(x, df=3)
plot(x, f, xlab="x", ylab="density", type="l", lwd=5)
segments(gene1$stat, 0, gene1$stat, dt(gene1$stat, df=3),
col=3, lwd=5)
segments(-gene1$stat, 0, -gene1$stat, dt(gene1$stat, df=3),
col=2, lwd=5)
segments(x[abs(x)>abs(gene1$stat)], 0,
x[abs(x)>abs(gene1$stat)], f[abs(x)>abs(gene1$stat)],
col=4, lwd=1)
> gene1
One Sample t-test
data: M[1, ]
t = 0.7433, df = 3, p-value = 0.5112
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-1.771051 2.850409
sample estimates:
mean of x
0.539679
Module 6: Hypothesis testing
p–value
bioinformatics.ca
More plots
Task:
Explore alternative shading.
Module 6: Hypothesis testing
bioinformatics.ca
example – HIV data
Is gene 4 differentially expressed?
Is the mean log ratio equal to zero?
# One sample t-test
data <- log(read.table(file="hiv.raw.data.24h.txt",
sep="\t", header=TRUE))
# Compute the log ratios
M <- (data[, 1:4] - data[, 5:8])
gene4 <- t.test(M[4, ], mu=0)
x <- seq(-4, 4, 0.1)
f <- dt(x, df=3)
plot(x, f, xlab="x", ylab="density", type="l", lwd=5)
segments(gene4$stat, 0, gene4$stat, dt(gene4$stat, df=3),
col=3, lwd=5)
segments(-gene4$stat, 0, -gene4$stat, dt(gene4$stat, df=3),
col=2, lwd=5)
segments(x[abs(x)>abs(gene4$stat)], 0,
x[abs(x)>abs(gene4$stat)], f[abs(x)>abs(gene4$stat)],
col=4, lwd=1)
> gene4
One Sample t-test
data: M[4, ]
t = 3.2441, df = 3, p-value = 0.0477
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
0.01854012 1.93107647
sample estimates:
mean of x
0.9748083
Module 6: Hypothesis testing
p–value
bioinformatics.ca
what is a p–value?
a) A measure of how much evidence we have against the
alternative hypothesis.
b) The probability of making an error.
c) Something that biologists want to be below 0.05 .
d) The probability of observing a value as extreme or more
extreme by chance alone.
e) All of the above.
Module 6: Hypothesis testing
bioinformatics.ca
two–sample t–test
Test if the means of two distributions are the same.
The datasets yi1, ..., yin are independent and normally distributed
with mean μi and variance σ2, N (μi,σ2), where i=1,2.
In addition, we assume that the data in the two groups are
independent and that the variance is the same.
H 0 : m1 = m 2
Module 6: Hypothesis testing
H1 : m1 ¹ m 2
bioinformatics.ca
two–sample t–test
Module 6: Hypothesis testing
bioinformatics.ca
example revisited – HIV data
Is gene 1 differentially expressed?
Are the means equal?
# Two sample t-test
data <- log(read.table(file="hiv.raw.data.24h.txt",
sep="\t", header=TRUE))
gene1 <- t.test(data[1,1:4], data[1,5:8], var.equal=TRUE)
x <- seq(-4, 4, 0.1)
f <- dt(x, df=6)
plot(x, f, xlab="x", ylab="density", type="l", lwd=5)
segments(gene1$stat, 0, gene1$stat, dt(gene1$stat, df=6),
col=3, lwd=5)
segments(-gene1$stat, 0, -gene1$stat, dt(gene1$stat, df=6),
col=2, lwd=5)
segments(x[abs(x)>abs(gene1$stat)], 0,
x[abs(x)>abs(gene1$stat)], f[abs(x)>abs(gene1$stat)],
col=4, lwd=1)
> gene1
Two Sample t-test
data: data[1, 1:4] and data[1, 5:8]
t = 0.6439, df = 6, p-value = 0.5434
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.511041 2.590398
sample estimates:
mean of x mean of y
2.134678 1.594999
Module 6: Hypothesis testing
p–value
bioinformatics.ca
example revisited – HIV data
Is gene 4 differentially expressed?
Are the means equal?
# Two sample t-test
data <- log(read.table(file="hiv.raw.data.24h.txt",
sep="\t", header=TRUE))
gene4 <- t.test(data[4,1:4], data[4,5:8], var.equal=TRUE)
x <- seq(-4, 4, 0.1)
f <- dt(x, df=6)
plot(x, f, xlab="x", ylab="density", type="l", lwd=5)
segments(gene4$stat, 0, gene4$stat, dt(gene4$stat, df=6),
col=3, lwd=5)
segments(-gene4$stat, 0, -gene4$stat, dt(gene4$stat, df=6),
col=2, lwd=5)
segments(x[abs(x)>abs(gene4$stat)], 0,
x[abs(x)>abs(gene4$stat)], f[abs(x)>abs(gene4$stat)],
col=4, lwd=1)
> gene1
Two Sample t-test
data: data[4, 1:4] and data[4, 5:8]
t = 2.4569, df = 6, p-value = 0.04933
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.003964227 1.945652365
sample estimates:
mean of x mean of y
7.56130 6.588321
Module 6: Hypothesis testing
p–value
bioinformatics.ca
t–test assumptions
Normality: The data need to be Normal. If not, one can use a
transformation or a non-parametric test. If the sample size is
large enough (n>30), the t-test will work just fine (CLT).
Independence: Usually satisfied. If not independent, more
complex modeling is required.
Independence between groups: In the two sample t- test, the
groups need to be independent. If not, one can use a paired ttest.
Equal variances: If the variances are not equal in the two
groups, use Welch's t-test (default in R).
Module 6: Hypothesis testing
bioinformatics.ca
non–parametric tests
Non-parametric tests constitute a flexible alternative to
t-tests if you don't have a model of the distribution.
In cases where a parametric test would be appropriate,
non-parametric tests have less power.
Several non parametric alternatives exist e.g. the
Wilcoxon and Mann-Whitney tests.
Module 6: Hypothesis testing
bioinformatics.ca
Wilcoxon test principle
o <- order(M[,1])
plot(M[o,1], col=M[o,2])
For each observation in a, count the
number of observations in b that
have a smaller rank.
The sum of these counts is the test
statistic.
wilcox.test(M[1:n,1], M[(1:n)+n,1])
Module 6: Hypothesis testing
bioinformatics.ca
permutation test
A p-value characterizes where an observation lies with
reference to the distribution of our statistics under the null
hypothesis.
How can we estimate the null distribution?
In the two sample case, to simulate the null distribution, one
could simply randomly permute the group labels and
recompute the statistics.
Repeat this for a (sufficienty large) number of permutations and
compute the number of times you randomly observed a value as
extreme or more extreme than the observation of interest.
Module 6: Hypothesis testing
bioinformatics.ca
permutation test
For data that has multiple "categories" associated with each
observation:
• Select a statistic (e.g. mean difference, t statistic)
• Compute the statistic for the observation of interest t.
• For a number of permutations N p
• Randomly permute the labels and compute the associated statistic ti0
• Count how often the statistic exceeds the observation
•
p(t) =
#
(
0
ti
>t
)
Np
Module 6: Hypothesis testing
bioinformatics.ca
the Bootstrap
The basic idea is to resample the
data we have observed and compute
a new value of the statistic/estimator
for each resampled data set.
Then one can assess the estimator
by looking at the empirical
distribution across the resampled
data sets.
Module 6: Hypothesis testing
set.seed(100)
x <- rnorm(15)
muHat <- mean(x)
sigmaHat <- sd(x)
Nrep <- 100
muHatNew <- rep(0, Nrep)
for(i in 1:Nrep) {
xNew <- sample(x, replace=TRUE)
muHatNew[i] <- median(xNew)
}
se <- sd(muHatNew)
muHat
se
bioinformatics.ca
statistical "power"
The power of a statistical test is the probability that the test will reject the null
hypothesis when the null hypothesis is false (i.e. that it will not make a Type II
error, or a false negative decision). As the power increases, the chances of
a Type II error occurring decrease. The probability of a Type II error occurring
is referred to as the false negative rate (β). Therefore power is equal to 1 − β,
which is also known as the sensitivity.
Power analysis can be used to calculate the minimum sample size required
so that one can be reasonably likely to detect an effect of a given size. Power
analysis can also be used to calculate the minimum effect size that is likely to
be detected in a study using a given sample size. In addition, the concept of
power is used to make comparisons between different statistical testing
procedures: for example, between a parametric and a nonparametric test of
the same hypothesis.
From Wikipedia – Statistical_Power
Module 6: Hypothesis testing
bioinformatics.ca
One sample t-test – power calculation
1 sample t-test:
If the mean is μ0, t follows a t-distribution with n-1 degrees of
freedom.
If the mean is not μ0, t follows a non central t-distribution with
n-1 degrees of freedom and noncentrality parameter
(μ1-μ0) x (s/√n).
Module 6: Hypothesis testing
bioinformatics.ca
Power, error rates and decision
Power calculation in R:
> power.t.test(n = 5, delta = 1, sd=2,
alternative="two.sided", type="one.sample")
One-sample t test power calculation
n=5
delta = 1
sd = 2
sig.level = 0.05
power = 0.1384528
alternative = two.sided
Other tests are available – see ??power.
Module 6: Hypothesis testing
bioinformatics.ca
Power, error rates and decision
PR(False Negative)
PR(Type II error)
μ0 μ 1
PR(False Positive)
PR(Type I error)
Module 6: Hypothesis testing
bioinformatics.ca
Power, error rates and decision
Module 6: Hypothesis testing
bioinformatics.ca
multiple testing
Single hypothesis testing
• Fix the False Positive error rate (eg. α = 0.05).
• Minimize the False Negative (maximize sensitivity)
This is what traditional testing does.
What if we perform many tests at once? Does this
affect our False Positive rate?
Module 6: Hypothesis testing
bioinformatics.ca
multiple testing
With high-throughput methods, we usually look at a very large
number of decisions for each experiment. For example, we ask
for every gene on an array whether it is significantly up- or
downregulated.
This creates a multiple testing paradox. The more data we
collect, the harder it is for every observation to appear
significant.
Therefore:
• We need ways to assess error probability in multiple testing
situations correctly;
• We need approaches that address the paradox.
Module 6: Hypothesis testing
bioinformatics.ca
multiple testing example
1000 t-tests, all null hypotheses are true (μ = 0).
For one test, Pr of a False Positive is 0.05.
For 1000 tests, Pr of at least one False Positive is 1–(1–0.05)1000 ≈ 1
> set.seed(100)
> y <- matrix(rnorm(100000), 1000, 5)
> myt.test <- function(y){
+ t.test(y, alternative="two.sided")$p.value
+}
> P <- apply(y, 1, myt.test)
> sum(P<0.05)
[1] 44
Module 6: Hypothesis testing
bioinformatics.ca
FWER
The FamilyWise Error Rate is the probability of having at least
one False Positive (making at least one type I error) in a "family" of
observations.
Example: Bonferroni multiple adjustment.
p̃g = N x pg
If p̃g ≤ α then FWER ≤ α
This is simple and conservative, but there are many other
(more powerful) FWER procedures.
Module 6: Hypothesis testing
bioinformatics.ca
False Discovery Rate (FDR)
The FDR is the proportion of False Positives among the genes
called differentially expressed (DE).
Order the p-values for each of N observations:
p(1) ≤ . . . ≤ p(i ) ≤ . . . ≤ p(N)
Let k be the largest i such that p(i ) ≤ i / N x α
... then the FDR for genes 1 ... k is controlled at α.
Hypotheses need to be independent!
FDR: Benjamini and Hochberg (1995)
Module 6: Hypothesis testing
bioinformatics.ca
FDR example
# FDR
set.seed(100)
N <- 10000
alpha <- 0.05
y1 <- matrix(rnorm(9000*4, 0, 1), 9000, 4)
y2 <- matrix(rnorm(1000*4, 5, 1), 1000, 4)
y <- rbind(y1, y2)
myt.test <- function(y){
t.test(y, alternative="two.sided")$p.value
}
P <- apply(y, 1, myt.test)
sum(P<alpha)
Psorted <- sort(P)
plot(Psorted, xlim=c(0, 1000), ylim=c(0, .01))
abline(b=alpha/N, a=0, col=2)
p <- p.adjust(P, method="bonferroni")
sum(p<0.05)
p <- p.adjust(P, method="fdr")
sum(p<0.05)
# Calculate the true FDR
sum(p[1:9000]<0.05)/sum(p<0.05)
Module 6: Hypothesis testing
bioinformatics.ca
FDR – applied to the HIV dataset
# Lab on HIV data
data <- log(read.table(file="hiv.raw.data.24h.txt", sep="\t", header=TRUE))
M <- data[, 1:4]-data[, 5:8]
A <- (data[, 1:4]+data[, 5:8])/2
Bonferroni p-values < 0.05
# Here I compute the mean over the four replicates
M.mean <- apply(M, 1, "mean")
A.mean <- apply(A, 1, "mean")
n <- nrow(M)
# Basic normalization
for(i in 1:4)
M[, i] <- M[, i]-mean(M[, i])
p.val <- rep(0, n)
for(j in 1:n)
p.val[j] <- t.test(M[j, ],
alternative = c("two.sided"))$p.value
p.val.tilde <- p.adjust(p.val, method="bonferroni")
plot(A.mean, M.mean, pch=".")
points(A.mean[p.val<0.05], M.mean[p.val<0.05], col=3)
points(A.mean[p.val.tilde<0.05], M.mean[p.val.tilde<0.05], col=4)
Module 6: Hypothesis testing
bioinformatics.ca
FDR – applied to the HIV dataset
FDR < 0.1
p.val.tilde <- p.adjust(p.val, method="fdr")
plot(A.mean, M.mean, pch=".")
points(A.mean[p.val<0.05], M.mean[p.val<0.05], col=3)
points(A.mean[p.val.tilde<0.1], M.mean[p.val.tilde<0.1], col=2)
Module 6: Hypothesis testing
bioinformatics.ca
FDR – applied to the HIV dataset
M.sd <- apply(M, 1, "sd")
FDR < 0.1
plot(M.mean, M.sd)
points(M.mean[p.val.tilde<0.1], M.sd[p.val.tilde<0.1], col=2)
The samples included
under the FDR criterion are
predominantly
characterized by a small
variance.
Is this biologically meaningful?
Module 6: Hypothesis testing
bioinformatics.ca
t–test revisited
1 sample t-test statistic:
tg =
Mg
Sg
R
If the mean is zero, tg approximately follows a t-distribution with R – 1
degrees of freedom.
Problem: for many genes S is small.
t g'
Mg
=
(Sg + c)
R
Solution: use a modified t-statistic.
(Positive constant)
Module 6: Hypothesis testing
bioinformatics.ca
SAM
SAM (Significance Analysis of Microarrays) is a statistical
technique to find significant expression changes of genes in
microaray experiments.
The input is an expression profile. SAM measures the
strength of the association of the expression value and the
conditions of the expression profile.
SAM employs a modified t-statistic that is more stable if
the number of conditions is small.
False Discovery Rates are estimated through permutations.
library(samr)
?samr
?SAM
Module 6: Hypothesis testing
bioinformatics.ca
SAM
FDR 10%
Module 6: Hypothesis testing
bioinformatics.ca
LIMMA
Exercise: perform the same analysis with LIMMA, a microarray
analysis package in the bioconductor project.
### Limma ###
source("http://bioconductor.org/biocLite.R")
biocLite("limma")
library(limma)
# Read about limma
limmaUsersGuide
## Compute necessary parameters (mean and standard deviations)
fit <- lmFit(M)
## Regularize the t-test
fit <- eBayes(fit)
## Default adjustment is BH (FDR)
topTable(fit, p.value=0.05, number=100)
Module 6: Hypothesis testing
bioinformatics.ca
summary
Sample
size
Number
of tests
p=1
p>1
n < 30
non-parametric
t-test/F- test
regularized
t-test/F-test
(e.g. SAM, limma)
+ multiple testing
n ≥ 30
t-test, F- test
t-test, F- test
+ multiple testing
Multiple testing:
If hypotheses are independent or weakly dependent use an FDR correction,
otherwise use Bonferroni's FWER.
For more complex hypotheses, try an ANOVA (p=1) or limma (p>1).
Module 6: Hypothesis testing
bioinformatics.ca
From here ...
Get a book. (e.g. Peter Dalgaard, Introductory
Statistics with R is available online through UofT library)
Simulate your data. (Don't just use the
packaged functions.)
Have fun.
Module 6: Hypothesis testing
bioinformatics.ca
[email protected]
Module 6: Hypothesis testing
bioinformatics.ca