Building risk prediction models - with a focus on Genome-Wide Association Studies. Charles Kooperberg

Transcription

1 Building risk prediction models - with a focus on Genome-Wide Association Studies

2 Risk prediction models Based on data: (D i, X i1,..., X ip ) i = 1,..., n we like to fit a model P(D = 1 X 1,..., X p ) = f (X 1,..., X p ), and evaluate how good the model is. Issues: Which X to use (model selection). What form to use for f (non-parametric(?) regression). Selection of smoothing/control parameters. Estimation of coefficients in the final model. Unbiased (?) evaluation of the model.

3 What form to use for f Not discussed in this talk! There are many (non-parametric) regression methods. Quite a few of these methods will involve variable selection, and/or selection of some hyper-parameters, in addition to fitting a regression model. In my experience often (not always) when there are many predictors linear models work (almost) as good as more complicated models (trees, splines, interactions... ), and they are often easier to explain, and come with less variance.

4 Selection of predictors. Selection of predictors on the same data as training and/or evaluating models can (sometimes severely) influence your results. In different ways. Be aware! Questions to ask - If certain markers are selected for the prediction, because they are the most significant ones, as reported in the literature, was the data you have in hand part of the data that was used to select those markers? Can you resist the temptation to use your test-set for anything until you have selected your final model. Are you sure that the subjects in your test data have not been used for this selection mentioned in the previous point. Remember, you can not even use it to select between the last few models! Using the same data to evaluate your model as is part of your cross-validation procedure biases your results. Carefully examine which steps influence each other.

5 In an ideal world you have loads of data 1 Selection data to select which predictors to use. 2 Training data to select control parameters (like λ in the Lasso) and parameters. You may need to do cross-validation to select λ, and then refit using all training data to fit the parameters. Alternatively, you may want separate data for Training and Validation. 3 Evaluation or Test data to evaluate your rule that has not been used before. If you don t have enough data, you sometimes can do a second level of cross-validation to effectively increase your sample size. The training and selection data can sometimes be combined, if you careful of what you do on which data.

6 Risk prediction models for GWAS Combine significant SNPs and environmental factors to predict risk of a disease. Do not worry about cause and effect of SNPs and these factors, as the goal is prediction. Do not worry about the form of the predictors: a black box is fine.

7 Where to worry about? Model selection: while there are many SNPs that can be significant (or not), there are many more possible models. We need efficient strategies to select models, as well as fair ways to compare them. Model evaluation: after all selection we also want to evaluate the quality of the model. We need new data for that. On a practical level - it is getting increasingly easy to get GWAS data from other groups. However, true prediction models also need information on other risk factors. Harmonizing those is a lot of work!

8 Lasso, LARS, boosting The traditional way to fit a logistic model with predictions X 1,..., X p is maximum likelihood. This works well provided the predictors are independent, and p < n the sample size. The Lasso (Tibshirani) noted that for prediction it is often better that some of the coefficients are shrunk (maybe all the way down to 0). This can be achieved by maximizing l(x, D; β) λ k β k, where l(x, D; β) is the logistic likelihood with parameters β, and λ > 0 is a penalty parameter.

9 Lasso, LARS, boosting (cont) There are close relations between the Lasso and (some forms of) boosting, or other related stepwise methods. These relations are partly formalized using the LARS (Efron, Friedman, Hastie, Tibshirani) algorithm. Parameter selection is usually via cross-validation. (But we still need a test-set to evaluate how good the prediction is.) It should be noted that these methods select the best predictors, not necessarily the significantly associated variables.

10 Lasso and GWAS The code may be efficient. But it cannot deal efficiently with 100,000s of predictors, and be able to do comparative simulation studies. The natural approach would appear to be to filter at a particular α level and only consider those predictors. The clean approach is to select significant SNPs each time separately for each cross-validation run (for λ). Potentially, the dirty way increases bias and decreases variance.

11 WTCCC data 3000 common controls 2000 cases for seven diseases each (Coronary artery disease, type 1 diabetes, type 2 diabetes, Crohn s, rheumatoid arthritis, bipolar disorder, hypertension). Affymetrix 5.0 ( 500,000 SNPs). Carried out experiment for T1D, T2D, Crohn s. For each of the diseases divide data in training set of 3000 and test set of Apply various prediction methods.

12 Preprocessing Filter out all SNPs with MAF < 5%, missingness > 5%, HW control P < Fill in missing data using single (random) imputation. (Makes life much easier.) Many methods can use with large datasets, but 500,000 predictors - certainly for simulations - is too much. Thus, filter and select the top p predictors. How? Correlations between predictors can get very high - as high as some methods will need some prior filtering.

13 Five selection procedures For each procedure set part of data apart as test-set (we took 40%). 1 Cheat Select the best predictors using all data. Fit using Lasso. Select the best λ using cross-validation (CV) on training data. Refit with that λ on all training data. 2 No-Test Select the best predictors using training data. Fit using Lasso. Select the best λ looking at the test data. 3 Part-CV Select the best predictors using training data. Fit using Lasso. Select the best λ using CV on training data. Refit with λ on all training data. 4 Full-CV CV: Select best predictors on 9/10th of training data. Get score for each λ, and evaluate on remaining 1/10th of training data. Select the best λ. Select best predictors using training data. Refit with λ on all training data. 5 Wei et al Divide training data in two parts. Use one part to select best predictors. Use other part to find best λ using CV. Refit with λ on second part of training data.

14 Five selection procedures For each procedure set part of data apart as test-set (we took 40%). 1 Cheat Select the best Training predictors using all data. Fit using Test Lasso. Select the best λ using cross-validation (CV) on training data. Refit with that λ on all training data. 2 No-Test Select the best predictors using training data. Fit using Lasso. Select the best λ looking at the test data. 3 Part-CV Select the best predictors using training data. Fit using Lasso. Select the best λ using CV on training data. Refit with λ on all training data. 4 Full-CV CV: Select best predictors on 9/10th of training data. Get score for each λ, and evaluate on remaining 1/10th of training data. Select the best λ. Select best predictors using training data. Refit with λ on all training data. 5 Wei et al Divide training data in two parts. Use one part to select best predictors. Use other part to find best λ using CV. Refit with λ on second part of training data. CV select lambda evaluate

15 Five selection procedures For each procedure set part of data apart as test-set (we took 40%). 1 Cheat Select the best Training predictors using all data. Fit using Test Lasso. Select the best λ using cross-validation (CV) on training data. Refit with that λ on all training data. Training Test 2 No-Test Select the best predictors using training data. Fit using Lasso. Select the best λ looking at the test data. 3 Part-CV Select the best predictors using training data. Fit using Lasso. Select the best λ using CV on training data. Refit with λ on all training data. 4 Full-CV CV: Select best predictors on 9/10th of training data. Get score for each λ, and evaluate on remaining 1/10th of training data. Select the best λ. Select best predictors using training data. Refit with λ on all training data. 5 Wei et al Divide training data in two parts. Use one part to select best predictors. Use other part to find best λ using CV. Refit with λ on second part of training data. CV select lambda evaluate select lambda evaluate

16 Five selection procedures For each procedure set part of data apart as test-set (we took 40%). 1 Cheat Select the best Training predictors using all data. Fit using Test Lasso. Select the best λ using cross-validation (CV) on training data. Refit with that λ on all training data. Training Test 2 No-TestSelect the best predictors using training data. Fit using Lasso. Select the best λ looking at the test data. 3 Part-CV Select the best predictors using training data. Fit Training CV Test using Lasso. Select the best λ using CV on training data. Refit with λ on all training data. 4 Full-CV CV: Select best predictors on 9/10th of training data. Get score for each λ, and evaluate on remaining 1/10th of training data. Select the best λ. Select best predictors using training data. Refit with λ on all training data. 5 Wei et al Divide training data in two parts. Use one part to select best predictors. Use other part to find best λ using CV. Refit with λ on second part of training data. CV select lambda evaluate select lambda evaluate select lambda evaluate

17 Five selection procedures For each procedure set part of data apart as test-set (we took 40%). 1 Cheat Select the best Training predictors using all data. Fit using Test Lasso. Select the best λ using cross-validation (CV) on training data. Refit with that λ on all training data. Training Test 2 No-Test Select the best predictors using training data. Fit using Lasso. Select the best λ looking at the test data. 3 Part-CV Select the best predictors using training data. Fit Training CV Test using Lasso. Select the best λ using CV on training data. Refit with λ on all training data. 4 Full-CV CV: Select best predictors on 9/10th of training data. Training CV Test Get score for each λ, and evaluate on remaining 1/10th of training data. Select the best λ. Select best predictors using training data. Refit with λ on all training data. 5 Wei et al Divide training data in two parts. Use one part to select best predictors. Use other part to find best λ using CV. Refit with λ on second part of training data. CV select lambda evaluate select lambda evaluate select lambda evaluate select lambda evaluate

18 Five selection procedures For each procedure set part of data apart as test-set (we took 40%). 1 Cheat Select the best Training predictors using all data. Fit using Test Lasso. Select the best λ using cross-validation (CV) on training data. Refit with that λ on all training data. Training Test 2 No-Test Select the best predictors using training data. Fit using Lasso. Select the best λ looking at the test data. 3 Part-CV Select the best predictors using training data. Fit Training CV Test using Lasso. Select the best λ using CV on training data. Refit with λ on all training data. 4 Full-CV CV: Select best predictors on 9/10th of training data. Training CV Test Get score for each λ, and evaluate on remaining 1/10th of training data. Select the best λ. Select best predictors using training data. Refit with λ on all training data. 5 Wei et al Divide training data in two parts. Use one part to select best predictors. Use other part to find best λ using CV. Refit with λ on second part of training data. CV Selection Training CV Test select lambda evaluate select lambda evaluate select lambda evaluate select lambda evaluate select lambda evaluate

19 Crohn s disease - Lasso Approach Cheat No-Test Part-CV Full-CV Filter All Training Training CV Parameters CV Test CV CV

20 Crohn s - average log-likelihood log likelihood training approach 1 test approach 1 training approach 3 test approach 3 training approach 4 test approach number of SNPs considered

21 Number of SNPs actually used GLM 17 Fitting the best p predictors; select p like λ. filtered GLM 22 Fitting the best p predictors; remove SNPs with R 2 > 0.9, select p like λ. stepwise GLM 23 Stepwise selection using BIC. Lasso 10 top SNPs considered 6 25 top SNPs considered top SNPs considered top SNPs considered top SNPs considered top SNPs considered top SNPs considered top SNPs considered 177

22 Crohn s average log likelihood training log lik test log lik training AUC test AUC GLM filtered GLM stepwise GLM AUC number of SNPs considered

23 ROC - Crohn s Number of SNPs considered

24 SNPs used - Crohn s BIC AIC glm filtered lasso: number of top SNPs considered SNP not used SNP used SNP used

25 Approach 5 - Wei et al. AJHG 2013

26 Approach 5 - Wei et al. AJHG 2013 Selection set 13, 273 Training set α 13, 273 Test set 13, 273 Not used (1 α) 13, 273

27 Verify results on another GWAS NIDDK Crohn s disease GWAS Illumina 300K 792 cases, 932 controls Refit selected models on complete WTCCC data. Impute the essential SNPs in the NIDDK data using MACH. Use ten probability samples, average results. Apply model to NIDDK data. Adjust intercept of logistic model to correct for different case/control ratio.

28 This is a high bar Different platform: we have to impute > 90% of the SNPs. Different populations. Different continents. No information whether disease adjudication is comparable.

29 AUC - NIDDK and WTCCC comparison AUC NIDDK WTCCC number of SNPs considered

30 Cross-study experiment fraction that is case WTCCC all as train NIDDK test fitted probability

31 Conclusions It is possible to develop prediction models with moderate predictive power using GWAS data. These predictive models produce results that are reproducible on other GWAS studies You have to be honest in cross-validation. Using more SNPs than are identified as significant helps. A shrinkage method like the Lasso helps.

32 References/Thanks Thanks: Michael LeBlanc, Valerie Obenchain, Li Hsu References: Kooperberg C, LeBlanc M, Obenchain V (2010). Risk prediction using genome-wide association studies. Genetic Epidemiology, 34, Wei Z, Wang W,... (2013). Large sample size, wide variant spectrum, and advanced machine-learning technique boost risk prediction for inflammatory bowel disease. American Journal of Human Genetics, 92,