Download Phase 1

Statistical, Computational, and
Informatics Tools for Biomarker
Analysis
Methodology Development at the
Data Management and Coordinating Center
of the
Early Detection Research Network
Early
Detection
Research
Network
EDRN ORGANIZATIONAL STRUCTURE
18 Laboratories
2 Laboratories
NIST
8 Centers
CDCP
Chair: Bernard Levin
Chair: David Sidransky
An “infrastructure”
for supporting
collaborative
research on
molecular, genetic
and other
biomarkers in human
cancer detection and
risk assessment.
Early
Detection
Research
Network
INFRASTRUCTURE
BIOREPOSITORY
• Specimens with matching controls and
epidemiological data
• Infrastructure to provide preneoplastic tissues:
- Prostate
- Lung
- Ovarian
- Colon
- Breast
Early
Detection INFRASTRUCTURE
Research
Network
LABORATORY CAPACITY
• Capability in high-throughput molecular and biochemical assays
• Ability to respond to evolving technologies for EDRN needs
• Extensive experience and scale-up ability in proteomics and
molecular assays
• Outstanding infrastructure for handling multiple assays and
validation requests
Early
Detection
Research
Network
INFRASTRUCTURE
DATA STORAGE AND MINING
• Outstanding track record in biomarker research
• Statistical and data mining technology
• Statistical and predictive models for multiple biomarkers
• Novel statistical methods to interpret high-throughput data
Early
Detection INFRASTRUCTURE
Research
Network
DATA EXCHANGE AND SHARING
•Improving informatics and information flow
Network web sites
public web site
secure web site
• Early Detection Research Network Exchange (ERNE)
• Standardizing of Data Reporting: CDEs Developed
Early Detection Research Network (EDRN)
INFORMATICS AND INFORMATION FLOW
EARLY
DETECTION
RESEARCH
NETWORK
COLLABORATION
How To Become an Associate Member
• Contact one of the EDRN Principal Investigators to serve as a
sponsor for an application. Three types of collaborative
opportunities are available:
Type A: Novel research ideas complementing EDRN
ongoing efforts; one year of funding at $100,000
Type B: Share tools, technology and resources, no time limit
Type C: Allow to participate in the EDRN Meetings and
Workshop
For details on how to apply, see http://www.cancer.gov/edrn
DMCC Statisticians
• Margaret Pepe, Lead of Methodology
Group
• Ziding Feng, Principal Investigator
• Yinsheng Qu
• Mary Lou Thompson
• Mark Thornquist
• Yutaka Yasui
Biomarker Lab Collaborators at Eastern
Virginia Medical School
• Bao-Ling Adam
• John Semmes
• George Wright
Focus of Presentation
• Design:
Phase Structure for Biomarker Research
• Analysis:
Statistical Methods for Biomarker
Discovery from High-Dimensional Data
Sets
Design:
Phase Structure for Biomarker Research
Three phase structure for therapeutic trials
well-established
Structure promotes coherent, thorough, efficient
development
Similar structure needs to be developed for
biomarker research
Biomarker Development
• Categorize process into 5 phases
• Define objectives for each phase
• Define ideal study designs, evaluation and
criteria for proceeding further
• Standardize the process to promote
efficiency and rigor
Figure 2. Phases of Biomarker Development
Preclinical
Exploratory
PHASE 1
Promising directions identified
Clinical Assay
and
Validation
PHASE 2
Clinical assay detects established
disease
Retrospective
Longitudinal
PHASE 3
Biomarker detects preclinical disease
and a “screen positive” rule defined
PHASE 4
Extent and characteristics of disease
detected by the test and the false
referral rate are identified
PHASE 5
Impact of screening on reducing
burden of disease on population is
quantified
Prospective
Screening
Cancer
Control
The Details of Study Design
• Specific Aims
• Subject/Specimen Selection
• Outcome measures
• Evaluation of Results
• Sample Size Calculations
• Limitations / Pitfalls
Specific Aims
Phase 1
Phase 2
• Identify leads for
potentially useful
biomarkers
• Determine the
sensitivity and
specificity or ROC
curve for the clinical
biomarker assay in
discriminating clinical
cancer from controls
• Prioritize these leads
Specimen Selection -- Cases
Phase 1
• Cancers that are
ultimately serious if
not treated early, but
treatable in early stage
• Spectrum of sub-types
• Collected at diagnosis
Phase 2: same criteria as
for phase 1
• Wide spectrum of cases
• Clinical specimen at
diagnosis
• From target screening
population
Specimen Selection -- Controls
Phase 1
Phase 2
• Non-cancer tissue same
organ same patient
• From potential target
population for screening
• Normal tissue non-cancer
patient
• Benign growth tissue noncancer patient
Outcome Measures
Phase 1
Phase 2
• True positive and False
positive rates (binary
result)
• Results of clinical
biomarker assay
• True positive rate at
threshold yielding
acceptable false positive
rate
• ROC curve
Evaluation of Results
Phase 1
Phase 2
• Algorithms select and
prioritize markers that best
distinguish tumor from
non-tumor tissue
• ROC curves
• Initial exploratory studies
need confirmation with
new validation specimens
• ROC regression to
determine if
characteristics of cases
and/or characteristics of
controls effect
biomarker’s
discriminatory capacity
Sample Size
Phase 1
Phase 2
• Should be large enough so
that very promising
biomarkers are likely to be
selected for phase 2
development
• Based on a confidence
intervals for the TPR or
FPR, or confidence
intervals for the ROC
curve at selected critical
points
Findings: Sample Size Estimation
• For phase 1 microarray experiments, use of
ROC curves is more efficient than
comparing means
• For phase 2 studies, equal numbers of cases
and controls is often not optimally efficient
• Sample size calculations and look-up tables
are now in EDRN website
1. Pepe et al. Phases of biomarker development for
early detection of cancer. Journal of the National
Cancer Institute 93(14):1054–61, 2001.
2. Pepe et al. “Elements of Study Design for
Biomarker Development” In Tumor Markers,
Diamandis, Fritsche, Lilja, Chan, and Schwartz ,
eds. AAAC Press, Washington, DC. 2002.
3. Pepe. “Statistical Evaluation of Diagnostic Tests
& Biomarkers” Oxford U. Press, 2003.
Selecting Differentially Expressed Genes
from Microarray Experiments
Lead: Margaret Pepe
Context
• gene expression arrays for nD tumor tissues and nC
normal tissues
• Yig = logarithm relative intensity at gene g for tissue i.
• for which genes are Yig different in some/most cases from the
normals?
• how many tissues, nD and nC, should be evaluated in these
experiments?
• illustrated with ovarian cancer data
Statistical Measures for Gene Selection
— typically use a two sample t-test for each gene
— we argue that sensitivity and specificity are more directly
relevant for cancer biomarker research.
— focus attention on high specificity (or high sensitivity)
— use the partial area under the ROC curve to rank genes,
instead of the t-test
Example
Gene Rank (among 100 genes)
gene #5
gene #97
t-test
10
4
partial AUC
3
31
gene 97
gene 5
diseased
1.0
diseased
15
5
10
Frequency
5
0
0
0
1
2
0
normal
1
2
3
4
5
6
7
normal
20
0.8
gene 5
0.6
gene 97
0.4
0.2
15
5
ROC(t) = P[YD > u]
20
10
5
0
0.0
0.0
0
0
1
2
0
1
2
3
4
5
6
0.2
0.4
0.6
7
t = P[YC > u]
0.8
1.0
Sample Sizes for Gene Discovery Studies
• traditional calculations based on statistical hypothesis testing
• These are exploratory studies, need new methods
• Propose to base calculations on the probability that a
differentially expressed gene will rank high among all genes
• Use computer simulation for sample size calculations
Table 3
Study power Pg {100|  k1} as a function of sample size using the ovarian cancer data as a
simulation model. Also shown is the power for the more stringent criterion Pg {100|  k1}.
True Ranking (k1)
< 10
< 20
(nD, nc)
(15, 15)
(25, 25)
(50, 50)
(100, 100)
.997
1.000
1.000
1.000
.982
.996
1.000
1.000
(15, 15)
(25, 25)
(50, 50)
(100, 100)
.960
1.000
1.000
1.000
.654
.928
1.000
1.000
Pg {100|  k1}
< 30
.934
.973
.994
.999
Pg {100|  k1}.
.120
.486
.836
.984
< 40
< 50
.893
.949
.987
.998
.850
.914
.968
.990
.016
.202
.638
.928
.000
.024
.206
.608
• with 50 tumor and 50 normal tissues we can be 83.6%
sure that the top 30 genes will rank in the top 100 in the
experiment.
• Pepe et al. Selecting differentially expressed
genes from microarray experiments.
Biometrics (in press)
Summary
• The method we developed for selecting
genes and calculating sample sizes are more
appropriate for the purpose of diagnosis and
early detection
Analysis:
Statistical Methods for Biomarker Discovery from
High-Dimensional Data Sets
• Method development motivated by SELDI data
from John Semmes/George Wright at Eastern
Virginia Medical School
• Data consist of protein intensities at tens of
thousands of mass/charge points on each of 297
individuals
• Developed three approaches to biomarker
discovery: wavelets, boosting decision tree, and
automated peak identification
The EVMS prostate cancer biomarker
project
• Prostate cancer patients:
N=99 early-stage
N=98 late-stage
• Normal controls
N=96
• Serum samples for proteomic analysis by Surface
Enhanced Laser Desorption/Ionization (SELDI)
• Goal: To discover protein signals that distinguish
cancers from normals
48,000 mass/charge points
(200K Da)
0
Intensity
2
4
6
8
An example of SELDI output
2000
3000
4000
5000
6000
Mass /Charge
7000
800
The design of the biomarker analysis
Normal
PCaearly
PCa-late
N=96
N=99
N=98
Training Data
167 PCa (84 early, 83 late)
vs.
81 Normal
Test Data
30 PCa
15 Normal
(Blinded)
Wavelet Analysis
Lead: Yinsheng Qu
Steps in the wavelet analysis:
• Represent original data plot with a set of
wavelets (dimension reduction)
• Determine those wavelets that distinguish
between subgroups (information criterion)
• Define discriminating functions based on
the distinguishing wavelets (Fisher
discrimination)
0.03
1.0
0.01
0.4
0.6
0.02
0.8
60
40
0
0.0
0.0
0.2
20
Original data
5000
10000
15000
20000
20000
40000
100000
M/Z
140000
180000
M/Z
1.0
0.4
0.010
0.6
0.020
0.8
60
40
0.0
0.0
0.2
20
0
Reconstructed signal
80000
0.030
M/Z
60000
5000
10000
M/Z
15000
20000
20000
40000
60000
M/Z
80000
100000
140000
M/Z
180000
0
20 40 60
R econ with 112 w c
0
20 40 60
R econ with 225 w c
0
20
40
60
R econ with 450 w c
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
2000
4000
6000
M/Z
8000
10000
0
20
40
60
Original data
Three Group Classification:
Normal, Cancer, BPH
12,352 mass spectrum data points, reduced to
3,420 Haar wavelet coefficients, of which
17 coefficients distinguish between the three cases.
2 classification functions generated.
Predicted:
Normal
Cancer
BPH
Normal
14
1
27
0
Truth:
Cancer BPH
0
7
3
0
8
Qu Y et al. Data reduction using discrete
wavelet transform in discriminant analysis
with very high dimension. Biometrics, in
press.
Boosted Decision Tree Method.
Lead: Yinsheng Qu/Yutaka Yasui
• This method combines multiple weak
learners into a very accurate classifier
• It can be used in cancer detection
• It can also be used in identification of tumor
markers
• Using this method we can separate controls,
BPH, and PCA without error in test set
Outline of boosting decision tree
• The combined classifier is a committee with the
decision stumps, the base classifiers, as its
members. It makes decisions by majority vote.
• The base classifiers are constructed on
weighted examples: the examples misclassified
will increase their weights on next round.
• The 2nd stump’s specialty is to correct the 1st
stump’s mistakes, and the 3rd stump’s specialty
is to correct the 2nd stump’s mistakes, and so
on.
• The combined classifier with dozens and even
hundreds of decision stumps will be accurate.
• Boosting technique is resistant to over
fitting.
Classifier 2: A boosted decision stump classifier with
21 peaks (potential markers)
normal
bph
cancer
sensitivity
specificity
# of peaks
minimal margin
Training set
Testing set
normal
bph
cancer
normal
bph
cancer
82
0
0
14
0
1
0
74
3
0
15
0
7
0
160
0
1
29
95.81%
96.67%
98.11%
96.67%
21 in 21 base classifiers
-0.2555
The Boosting procedure
•
•
•
•
Yi={cancer, normal}={1, -1}, fm(xi)={1, -1}
Initial weights (m=1), wi = 1 (i = 1, . . .,N).
Choose first peak and threshold c.
For m =1 to M: wi = wi exp{am I(incorrect)}
– where am = ln(1-err)/err) and err is
classification error rate at the current stage
– normalize the weights so they sum to N.
– choose a peak and c (i-th subject with weight wi)
the
• Final classifier: f(x) = sum(amfm(x)) over m=1 to
M. f(xi)> 0  i-th subject classified as cancer
When to stop iteration?
• minimal margin: minimum of yi f(xi) over all N
subjects
• The minimal margin in the training sample
measures how well the two classes are separated
by classifier.
• Even classifier reaches zero error on training
sample, if iteration still increases the minimal
margin --> improve prediction in future samples.
• Qu et al. 2002. Boosted Decision Tree
Analysis of SELDI Mass Spectral Serum
Profiles Discriminates Prostate Cancer from
Non-Cancer Patients. Clinical Chemistry. In press.
• Adam et al. 2002. Serum Protein
Fingerprinting Coupled with a Pattern
Matching Algorithm that Distinguishes
Prostate Cancer from Benign Prostate
Hyperplasia and Healthy Men. Cancer
Research. 62:3609-3614.
Summary
• Wavelets approach: Does not require peak
identification (black-box classification)
• Boosting decision tree: Requires peak
identification first. Useful for both
classification and protein mass
identification
Final Summary
• The methods developed in the past two years are
mainly for Phase 1&2 studies, reflecting the
current needs of EDRN.
• EDRN DMCC statisticians are working on key
design and analysis issues in early detection
research.
• More work remains to be done (e.g., In
classification, consider the mislabeling of Prostate
cancer by BPH; exam gene by environmental
interactions).