Download Powerpoint Slides - Iowa State University

A Discussion of False Discovery Rate and
the Identification of Differentially Expressed
Gene Categories in Microarray Studies
Dan Nettleton
Iowa State University
Ames, Iowa
August 8, 2007
1
Myostatin Knockout Mice vs. Wild Type
Belgian Blue
cattle have a
mutation in the
myostatin gene.
2
Affymetrix GeneChips on 5 Mice per Genotype
M
WT
M
WT
WT
M
WT
M
WT
M
3
The Dataset
Gene
ID
Wild Type
Mutant
p-value
1
4835.8
4578.2
4856.3
4483.7
4275.3
4170.7
3836.9
3901.8
4218.4
4094.0
p1
2
153.9
161.0
139.7
173.0
160.1
180.1
265.1
201.2
130.8
130.7
p2
3
3546.5
3622.7
3364.3
3433.6
2757.2
3346.9
2723.8
2892.0
3021.3
2452.7
p3
4
711.3
717.3
776.6
787.5
750.3
910.2
813.3
687.9
811.1
695.6
p4
5
126.3
178.2
114.5
158.7
157.3
231.7
147.0
102.8
157.6
146.8
p5
6
4161.8
4622.9
3795.7
4501.2
4265.8
3931.3
3327.6
3726.7
4003.0
3906.8
p6
7
419.3
555.3
509.6
515.5
488.9
426.6
425.8
500.8
347.8
580.3
p7
8
2420.7
2616.1
2768.7
2663.7
2264.6
2379.7
2196.2
2491.3
2710.0
2759.1
p8
9
321.5
540.6
471.9
348.2
356.6
382.5
375.9
481.5
260.6
515.7
p9
10
1061.4
949.4
1236.8
1034.7
976.8
1059.8
903.6
1060.3
960.1
1134.5
p10
11
1293.3
1147.7
1173.8
1173.9
1274.2
1062.8
1172.1
1113.0
1432.1
1012.4
p11
12
336.1
413.5
425.2
462.8
412.2
391.7
388.1
363.7
310.8
404.6
p12
13
..
.
325.2
..
.
278.9
..
.
242.8
..
.
255.6
..
.
283.5
..
.
161.1
..
.
181.0
..
.
222.0
..
.
279.3
..
.
232.9
..
.
p13
22690
249.6
283.6
271.0
246.9
252.7
214.2
217.9
266.6
193.7
413.2
..
.
p
22690
4
A Standard Analysis
• Two-sample t-tests for each gene.
• Compute p-values by comparing t-statistics to a
t-distribution with 8 d.f.
• Use an adjustment for multiple testing to create
a list of genes declared to be differentially
expressed.
5
Number of Genes
Histogram of p-values
from the Two-Sample t-Tests
p-value
6
Example p-value Distributions
Two-Sample t-test of H0:μ1=μ2
n1=n2=5, variance=1
μ1-μ2=1
μ1-μ2=0.5
μ1-μ2=0
7
Number of Genes
Histogram of p-values
from the Two-Sample t-Tests
p-value
8
False Discovery Rate (FDR)
• FDR is an error measure that can be useful for multiple
testing problems encountered in microarray experiments.
• FDR was introduced by Benjamini and Hochberg (1995)
and is formally defined as follows:
R = # rejected null hypotheses when conducting m tests
V = # of type I errors (false discoveries)
FDR=E(Q) where Q=V/R if R>0 and Q=0 otherwise.
9
A Conceptual Description of FDR
• Suppose a scientist conducts many independent
microarray experiments.
• For each experiment, the scientist uses a method for
producing a list of genes declared to be differentially
expressed.
• For each list consider the ratio of the number of false
positive results to the total number of genes on the list
(set this ratio to 0 if the list contains no genes).
• The FDR is approximated by the average of the ratios
described above.
10
A Conceptual Description of FDR (continued)
• Some of the gene lists may contain a high
proportion of false positive results and yet the
method used may still control FDR at a given
level.
• It is the average performance across repeated
experiments that matters.
11
Number of Genes Declared to be Differentially
Expressed for Various Estimated FDR Levels
FDR Number of Genes P-Value Threshold
0.01
8
0.000003
0.05
313
0.000900
0.10
748
0.004339
0.15
1465
0.012730
0.20
2143
0.024909
FDR estimated using the method
of Storey and Tibshirani (2003).
12
Using Information about Genes to Interpret the
Results of Microarray Experiments
• Based on a large body of past research, some
information is known about many of the genes
represented on a microarray.
• The information might include tissues in which a gene is
known to be expressed, the biological process in which a
gene’s protein is known to act, or other general or quite
specific details about the function of the protein
produced by a gene.
• By examining this information in concert with the results
of a microarray experiment, biologists can often gain a
greater understanding of their microarray experiments.
13
Gene Ontology (GO) Terms
• GO terms provide one example of information that is
available about genes.
• The GO project provides three ontologies (structured
controlled vocabularies) that describe a gene’s
1. Biological Processes,
2. Cellular Components, and
3. Molecular Functions.
14
Gene Ontology (GO) Terms
• Each gene may be associated with 0 or more
GO terms in a given ontology.
• The GO terms in each ontology have varying
levels of specificity.
• The GO terms in each ontology can be
organized in a directed acyclic graph (DAG)
where each node represents a term and arrows
point from specific terms to more general terms.
15
Portion of the Biological Processes Ontology
Shown in a DAG
Alcohol Metabolic
Process
Energy Derivation by Oxidation
of Organic Compounds
Carbohydrate Metabolic
Process
Generation of Precursor
Metabolites and Energy
Cellular Metabolic
Process
Macromolecule
Metabolic Process
Cellular
Process
Primary Metabolic
Process
Metabolic
Process
Biological
Process
16
Constructing Gene Categories from GO Terms
• The set of genes associated with any particular GO term
could be considered as a category or gene set of interest
for subsequent testing.
• For example, we might ask if genes that are associated
with the Molecular Function term muscle alpha-actinin
binding are affected by a treatment of interest.
• We could simultaneously query many groups, general
and specific, to better understand the impact of
treatment on expression.
17
Simultaneous Testing of Multiple Categories with
Various Levels of Specificity
muscle alpha-actinin
binding
alpha-actinin
binding
beta-actinin
binding
actinin
binding
myosin
binding
ATPase
binding
cytoskeletal protein
binding
RNA polymerase
core enzyme binding
enzyme
binding
protein
binding
binding
molecular
function
18
Some Formal Methods for Testing Gene
Categories with Microarray Data
• Fisher’s exact test on lists of gene declared to be
differentially expressed (DDE)
• Gene Set Enrichment Analysis (GSEA)
• Significance Analysis of Function and Expression
(SAFE)
• Pathway Level Analysis of Gene Expression (PLAGE)
• Domain Enhanced Analysis (DEA)
• Many others appearing and soon to appear
19
Number of Genes Declared to be Differentially
Expressed for Various Estimated FDR Levels
FDR Number of Genes P-Value Threshold
0.01
8
0.000003
0.05
313
0.000900
0.10
748
0.004339
0.15
1465
0.012730
0.20
2143
0.024909
FDR estimated using the method
of Storey and Tibshirani (2003).
20
Are genes of category X overrepresented
among the genes
declared to be differentially expressed?
Gene of
Category X?
Declared to be yes
Differentially
Expressed?
no
yes
no
50
250
300
50
19650
19700
100
19900
20000
Highly significant overrepresentation according
to a chi-square test or Fisher’s exact test.
21
Problems with Chi-Square or Fisher’s Exact Test
for Detecting Overrepresentation
• The outcome of the overrepresentation test depends on
the significance threshold used to declare genes
differentially expressed.
• Functional categories in which many genes exhibit small
changes may go undetected.
• Genes are not independent, so a key assumption of the
chi-square and Fisher’s exact tests is violated.
• Information in the multivariate distribution of genes in a
category is not utilized.
22
Gene 2 Expression
Gene 2 Expression
Advantage of a Multivariate Approach
Gene 1 Expression
Gene 1 Expression
23
Multiresponse Permutation Procedure (MRPP)
• Mielke and Berry, (2001). Permutation Methods:
A Distance Function Approach. Springer, N.Y.
• Nonparametric test for a difference among
multivariate distributions
• Test statistic based on within-group inter-point
distances
• P-value obtained by data permutation
24
Expression Measure for Gene 2
The MRPP Test: An Illustrative Example
For balanced data, test statistic is sum
of within-group inter-point distances.
Expression Measure for Gene 1
25
Expression Measure for Gene 2
The test statistic is computed for
all permutations of the data.
Expression Measure for Gene 1
26
Expression Measure for Gene 2
An Example Permutation
Expression Measure for Gene 1
27
Expression Measure for Gene 2
Test statistic will be larger for permutations
than for the original data
Permutation p-value = # ( Toriginal ≥ Tpermutation ) = 2
# permutations
Expression Measure for Gene 1
252
28
Portion of Directed Acyclic Graph
of GO Molecular Function Terms
29