Download Validation and Replication

Validation and Replication
Overview
Definitions of validation and replication
Difficulties and limitations
Working examples from our group and others
Why?
False positive results still occur…. even after stringent QC, data
pre-processing, complex analyses and alpha adjustments
The best ways of ensuring an observation is in fact real and
meaningful is to:
•
validate and replicate the findings
•
perform longitudinal and functional studies to determine the true
causal/biological effects
Validation vs. Replication
Validation
Verify that the methylation data generated are accurate and the
results are reliable
Ideally, by repeating the experiment in the same samples but
using different laboratory techniques
Several factors could result in erroneous data. For instance:
•
•
•
systematic errors associated with the laboratory methods
experimental design issues (e.g. cases and controls on separate plates)
handling errors (e.g. sample mix-ups)
Validation enables you to ensure the findings are due to true
biological variation and not some unknown experimental
artefact
Replication vs. Validation
Replication
Reproduce the findings in a independent dataset, i.e. different
samples
Replication enables:
•
•
•
•
verification of the findings in a different dataset
the findings to be generalised to the wider population
a more precise estimate of the findings to be measured
further exploration
The ideal scenario
Perform both
Validation proves the results are reliable but not necessarily
generalisable to the wider population
Replication, if successful, proves the results are generalisable
But, if unsuccessful, you will not know why
•
technical error in the first and/or second stage
•
lack of power in the second stage
•
subtle sample/phenotypic differences
•
quite simply, a false positive finding due to chance in the first stage
In reality
Its not always possible to do both
•
Epigenetic techniques are expensive
•
Sites of interest may not be feasible on certain platforms
•
Limited access to tissue samples
•
Limited access to similar phenotypic cohorts
•
Application of different study designs e.g. parent-offspring pairs,
monozygotic twins, longitudinal studies may not be possible
Any attempt at validation and/or replication is better than
nothing
Summary so far
Validation:
Verify that the methylation data generated are accurate and the
results are reliable
• same samples, different method
Replication:
Reproduce the findings in an independent dataset
• different samples
Validation and replication are not the same thing, but both are
valuable tools
Examples from our group
We have utilised a number of different processes:
Repeat the experiment in the same samples using a different methodology
Repeat the experiment in the same samples using a different source of tissue
but the same technique
Include extra samples to increase robustness
Assess different measures
(e.g. expression, methylation, SNP genotypes)
Independent replication i.e. different samples but same experimental method
and study design
Identify methylation differences associated with Leber’s
hereditary optic neuropathy
Example 1. Leber’s Hereditary Optic Neuropathy (LHON)
LHON is a common mitochondrial disorder characterised by loss of
central vision
Hypothesis: Oxidative stress arising from mitochondrial dysfunction
alters DNA methylation of the nuclear genome with consequences for
the regulation of gene expression
We measured DNA methylation of the nuclear genome using 27k array
to identify differences between those with LHON phenotype and
unaffected carriers
• Samples from four pedigrees from the North East of
England.
Identify methylation differences associated with Leber’s
hereditary optic neuropathy
UK family pedigrees with Leber’s
hereditary optic neuropathy
French family
pedigrees
Discovery
Validation
Validation/Replication
Replication
Blood samples
Blood samples
Blood samples
Independent
cohort
27k chip Identify
differentially
methylated CpG
sites (n=28)
Bisulphite
modification &
Pyrosequencing of 2
candidates (n=28)
Bisulphite
modification &
Pyrosequencing of 2
candidates (n=49)
Bisulphite
modification &
Pyrosequencing
2 CpG sites
selected to take
forward (p<0.05)
Methylation levels strongly
correlated (rho >0.6)
between techniques and
trends in association for
both genes (p<0.1)
With an additional 19
samples mainly from the
same families, one
candidate remained
associated (p=0.006) the
other did not (p>0.1)
Hannah Elliott,
ongoing
Postnatal growth and DNA methylation are associated with
differential gene expression of TACSTD2 and childhood fat mass
Example 2. Postnatal growth and DNA methylation are
associated with differential gene expression of TACSTD2
and childhood fat mass
microarray expression analysis to identify genes with differential expression in
preterm-born children defined as slow or rapid growers.
• Identify potential candidates for methylation analysis
Postnatal growth and DNA methylation are associated with
differential gene expression of TACSTD2 and childhood fat mass
CHILDREN BORN PRETERM: Newcastle Preterm birth cohort
Blood samples 11yrs
Saliva samples 11yrs
DNA
RNA
DNA
expression microarray
slow vs rapid postnatal
growth (n=20)
Bisulphite modification
Bisulphite modification
Validation of top hit
using Real time PCR
Pyrosequencing analysis
of candidate gene (n=94)
Pyrosequencing analysis
of candidate gene (n=68)
Analysis of relationship between methylation,
expression and phenotype at age 11y
Alix Groom et al, Diabetes 2012
Postnatal growth and DNA methylation are associated with
differential gene expression of TACSTD2 and childhood fat mass
CHILDREN BORN TERM: ALSPAC
Cord blood samples
Blood samples 7yrs
DNA
DNA
Bisulphite modification
Bisulphite modification
Pyrosequencing analysis of
candidate gene (n=173)
Pyrosequencing analysis
of candidate gene (n=178)
Analysis of relationship between methylation and
phenotype at age 9 and 15 years
Alix Groom et al, Diabetes 2012
Smoking and methylation
Example 3 (not from our group)
177 individuals from the population-based epidemiological ESTHER study:
current smokers, former smokers, and those who had never smoked
Illumina HumanMethylation 27K BeadChip
Smoking and Methylation
177 individuals from ESTHER
study
Further
discovery
Discovery
Validation
Replication
Blood samples
Blood samples
Blood samples
27k Chip Identify
differentially
methylated CpG
sites
Bisulphite modification
& Sequenom EpiTYPER
analysis of discovery
samples
Bisulphite modification
& Sequenom EpiTYPER
analysis of 328 nonoverlapping subjects
1 CpG site
selected to take
forward
Spearman correlation
between methods:
(rho =0.82)
Smokers still
hypomethylated at CpG site
(Psmoking = 1.07x10-28)
Pronounced association
with smoking remained
Looked at
methylation in
surrounding
regions using
Sequenom
EpiTYPER
79 samples from
the discovery study
Only CpG sites
immediately next to
the main hit were
associated with
smoking (41bp away)
Smoking and Methylation
…They then went on to test the same methylation site in a different cohort
(Better replication?)
• Sequenom EpiTYPER analysis
• This time looking at whether F2RL3 methylation was related to a
clinical outcome
1206 individuals from the KAROLA prospective cohort study
• Experienced acute coronary syndrome, myocardial infarction or
coronary intervention
• Active follow up over 8 years
Smoking and Methylation
Methylation at F2RL3 associated
with mortality in patients in this
cohort
! The methylation data (CpG_4)
reported in the main body of the
paper IS NOT the same CpG site
described in the original paper. This
CpG is “CpG_2” – see
supplementary data for results
The strongest signal from the first
round wasn’t the strongest
association when linked to clinical
outcome in a second cohort
Conclusions
Validation and replication are different
Ideally, attempt to do both
Plan for further functional work or analysis to identify true
causal/biological effects
If you can….
Do it!
References
Breitling LP et al., Eur Heart J. 2012 Apr 17:
Smoking, F2RL3 methylation, and prognosis in stable coronary heart disease
Breitling LP et al., Am J Hum Genet. 2011 Apr 8;88(4):450-7. Epub 2011 Mar 31:
Tobacco-smoking-related differential DNA methylation: 27K discovery and replication
Groom A et al., Diabetes 2012 Feb;61(2):391-400. Epub 2011 Dec 21:
Postnatal growth and DNA methylation are associated with differential gene expression of the TACSTD2
gene and childhood fat mass
Hirschhorn JN and Daly MJ. Nat Rev Genet. 2005 Feb;6(2):95-108:
Genome-wide association studies for common diseases and complex traits
Rakyan VK et al., Nat Rev Genet. 2011 Jul 12;12(8):529-41. doi: 10.1038/nrg3000:
Epigenome-wide association studies for common human diseases
Validation and Replication