Validation and Replication
Overview Definitions of validation and replication Difficulties and limitations Working examples from our group and others
Why? False positive results still occur. even after stringent QC, data pre-processing, complex analyses and alpha adjustments The best ways of ensuring an observation is in fact real and meaningful is to: validate and replicate the findings perform longitudinal and functional studies to determine the true causal/biological effects
Validation vs. Replication Validation Verify that the methylation data generated are accurate and the results are reliable Ideally, by repeating the experiment in the same samples but using different laboratory techniques Several factors could result in erroneous data. For instance: systematic errors associated with the laboratory methods experimental design issues (e.g. cases and controls on separate plates) handling errors (e.g. sample mix-ups) Validation enables you to ensure the findings are due to true biological variation and not some unknown experimental artefact
Replication vs. Validation Replication Reproduce the findings in a independent dataset, i.e. different samples Replication enables: verification of the findings in a different dataset the findings to be generalised to the wider population a more precise estimate of the findings to be measured further exploration
The ideal scenario Perform both Validation proves the results are reliable but not necessarily generalisable to the wider population Replication, if successful, proves the results are generalisable But, if unsuccessful, you will not know why technical error in the first and/or second stage lack of power in the second stage subtle sample/phenotypic differences quite simply, a false positive finding due to chance in the first stage
In reality Its not always possible to do both Epigenetic techniques are expensive Sites of interest may not be feasible on certain platforms Limited access to tissue samples Limited access to similar phenotypic cohorts Application of different study designs e.g. parent-offspring pairs, monozygotic twins, longitudinal studies may not be possible Any attempt at validation and/or replication is better than nothing
Summary so far Validation: Verify that the methylation data generated are accurate and the results are reliable same samples, different method Replication: Reproduce the findings in an independent dataset different samples Validation and replication are not the same thing, but both are valuable tools
Examples from our group We have utilised a number of different processes: Repeat the experiment in the same samples using a different methodology Repeat the experiment in the same samples using a different source of tissue but the same technique Include extra samples to increase robustness Assess different measures (e.g. expression, methylation, SNP genotypes) Independent replication i.e. different samples but same experimental method and study design
Identify methylation differences associated with Leber s hereditary optic neuropathy Example 1. Leber s Hereditary Optic Neuropathy (LHON) LHON is a common mitochondrial disorder characterised by loss of central vision Hypothesis: Oxidative stress arising from mitochondrial dysfunction alters DNA methylation of the nuclear genome with consequences for the regulation of gene expression We measured DNA methylation of the nuclear genome using 27k array to identify differences between those with LHON phenotype and unaffected carriers Samples from four pedigrees from the North East of England.
Identify methylation differences associated with Leber s hereditary optic neuropathy UK family pedigrees with Leber s hereditary optic neuropathy French family pedigrees Discovery Validation Validation/Replication Replication Blood samples Blood samples Blood samples Independent cohort 27k chip Identify differentially methylated CpG sites (n=28) Bisulphite modification & Pyrosequencing of 2 candidates (n=28) Bisulphite modification & Pyrosequencing of 2 candidates (n=49) Bisulphite modification & Pyrosequencing 2 CpG sites selected to take forward (p<0.05) Methylation levels strongly correlated (rho >0.6) between techniques and trends in association for both genes (p<0.1) With an additional 19 samples mainly from the same families, one candidate remained associated (p=0.006) the other did not (p>0.1) Hannah Elliott, ongoing
Postnatal growth and DNA methylation are associated with differential gene expression of TACSTD2 and childhood fat mass Example 2. Postnatal growth and DNA methylation are associated with differential gene expression of TACSTD2 and childhood fat mass microarray expression analysis to identify genes with differential expression in preterm-born children defined as slow or rapid growers. Identify potential candidates for methylation analysis
Postnatal growth and DNA methylation are associated with differential gene expression of TACSTD2 and childhood fat mass CHILDREN BORN PRETERM: Newcastle Preterm birth cohort Blood samples 11yrs Saliva samples 11yrs RNA DNA DNA expression microarray slow vs rapid postnatal growth (n=20) Bisulphite modification Bisulphite modification Validation of top hit using Real time PCR Pyrosequencing analysis of candidate gene (n=94) Pyrosequencing analysis of candidate gene (n=68) Analysis of relationship between methylation, expression and phenotype at age 11y Alix Groom et al, Diabetes 2012
Postnatal growth and DNA methylation are associated with differential gene expression of TACSTD2 and childhood fat mass CHILDREN BORN TERM: ALSPAC Cord blood samples Blood samples 7yrs DNA DNA Bisulphite modification Bisulphite modification Pyrosequencing analysis of candidate gene (n=173) Pyrosequencing analysis of candidate gene (n=178) Analysis of relationship between methylation and phenotype at age 9 and 15 years Alix Groom et al, Diabetes 2012
Smoking and methylation Example 3 (not from our group) 177 individuals from the population-based epidemiological ESTHER study: current smokers, former smokers, and those who had never smoked Illumina HumanMethylation 27K BeadChip
Smoking and Methylation 177 individuals from ESTHER study Further discovery Discovery Blood samples Validation Blood samples Replication Blood samples Looked at methylation in surrounding regions using Sequenom EpiTYPER 27k Chip Identify differentially methylated CpG sites Bisulphite modification & Sequenom EpiTYPER analysis of discovery samples Bisulphite modification & Sequenom EpiTYPER analysis of 328 nonoverlapping subjects 79 samples from the discovery study 1 CpG site selected to take forward Spearman correlation between methods: (rho =0.82) Smokers still hypomethylated at CpG site (P smoking = 1.07x10-28 ) Pronounced association with smoking remained Only CpG sites immediately next to the main hit were associated with smoking (41bp away)
Smoking and Methylation They then went on to test the same methylation site in a different cohort (Better replication?) Sequenom EpiTYPER analysis This time looking at whether F2RL3 methylation was related to a clinical outcome 1206 individuals from the KAROLA prospective cohort study Experienced acute coronary syndrome, myocardial infarction or coronary intervention Active follow up over 8 years
Smoking and Methylation Methylation at F2RL3 associated with mortality in patients in this cohort! The methylation data (CpG_4) reported in the main body of the paper IS NOT the same CpG site described in the original paper. This CpG is CpG_2 see supplementary data for results The strongest signal from the first round wasn t the strongest association when linked to clinical outcome in a second cohort
Conclusions Validation and replication are different Ideally, attempt to do both Plan for further functional work or analysis to identify true causal/biological effects If you can. Do it!
References Breitling LP et al., Eur Heart J. 2012 Apr 17: Smoking, F2RL3 methylation, and prognosis in stable coronary heart disease Breitling LP et al., Am J Hum Genet. 2011 Apr 8;88(4):450-7. Epub 2011 Mar 31: Tobacco-smoking-related differential DNA methylation: 27K discovery and replication Groom A et al., Diabetes 2012 Feb;61(2):391-400. Epub 2011 Dec 21: Postnatal growth and DNA methylation are associated with differential gene expression of the TACSTD2 gene and childhood fat mass Hirschhorn JN and Daly MJ. Nat Rev Genet. 2005 Feb;6(2):95-108: Genome-wide association studies for common diseases and complex traits Rakyan VK et al., Nat Rev Genet. 2011 Jul 12;12(8):529-41. doi: 10.1038/nrg3000: Epigenome-wide association studies for common human diseases
Validation and Replication