Is Grid or Cloud Computing Suitable for Normalization of Microarray Data and Resampling Approaches for SNP Data?

Transcription

1 1/ 16 Is Grid or Cloud Computing Suitable for Normalization of Microarray Data and Resampling Approaches for SNP Data? for Computational (Bio-)Statistics LMU Munich Institute for Medical Information Sciences, Biometry, and Epidemiology

2 2/ Project 1: Normalization of Microarray Data 2. Project 2: Resampling Approaches on SNP-Data 3. Outlook & Questions

3 Outline Normalization of Microarrays Resampling for SNP-Data 3/ 16 Project 1: Normalization of Microarray Data

4 Outline Normalization of Microarrays Resampling for SNP-Data Benefit of Addon-Normalization and Correct Normalization in Resampling Setups Evaluation of prediction models and the effect of normalization on the quality of prediction: strict separation of training and test set in resampling test set to be preprocessed separately possible approaches: each node performs one resampling step each node needs to be sent the whole AffyBatch-object or access to all.cel-files each node operates on small fraction of the microarrays (as in affypara) more communication between nodes during actual computation time (works for normalization step only) 4/ 16

5 Outline Normalization of Microarrays Resampling for SNP-Data Illustration - Effect of Normalization 5/ 16

6 Outline Normalization of Microarrays Resampling for SNP-Data Solution at the LRZ-cluster 6/ 16 Each node performs normalization step and subsequent classification step separately: host reads in.cel-files and broadcasts them to all nodes AffyBatch (faster than reading from hard disc) host sends training indices to each node different normalization methods and classification methods performed separately on each node host collects classification performances embarrassingly parallel scaled quite well

7 Outline Normalization of Microarrays Resampling for SNP-Data Speedup 7/ threads good speedup up to 125 workers computation time reduced from 124 hours to less than 90 minutes (125 workers) multicore (R) at HLRBII

8 Outline Normalization of Microarrays Resampling for SNP-Data Suitable Task for Grids/Clouds? Solution 1: Broadcast of AffyBatch on Grids/Clouds? requires 12GB RAM and more for large microarray studies (Can that be guaranteed on Grids or Clouds?) Solution 2: split data approach not realizable via broadcasts how to transfer respective parts of the data to the nodes? host reads in and sends them to the nodes nodes read from database themselves steady communication between nodes waiting times for slowest node in the grid/cloud 8/ 16

9 Normalization of Microarrays Resampling for SNP-Data Outlook & Questions 9/ 16 Project 2: Resampling Approaches on Genome-Wide SNP-Data

10 Normalization of Microarrays Resampling for SNP-Data Outlook & Questions Resampling on genome wide SNP-Data 10/ 16 2-step-approach: variable selection for all SNPs primarily univariate methods (filtering) wrapper approaches? regression/classification using selected SNPs only resembles classification/regression task on microarrays since only a small number of SNPs is selected construction of a single classifier computationally not very demanding

11 Normalization of Microarrays Resampling for SNP-Data Outlook & Questions 11/ 16 Possible Realization Filter-Step: each node could perform all gene selection steps for a small fraction of the data set no need to send whole data set to all nodes point-to-point communication or individual reading from data base? results collected by host Classification/Regression-Step: each node performs one resampling iteration using respective variables only heterogeneity and waiting times probably not problematic

12 Normalization of Microarrays Resampling for SNP-Data Outlook & Questions Illustration - Different Implementations 12/ 16

13 Resampling for SNP-Data Outlook & Questions 13/ 16 Outlook & Questions

14 Resampling for SNP-Data Outlook & Questions 14/ 16 Some questions: Are workers allowed to communicate with other workers? Is it possible to send or broadcast whole genome wide SNP data within a sensible time frame? Is there a guaranteed amount of RAM that can be specified? What to do if one node produces an error? Can the respective task be performed on another node without aborting the whole job? What is more efficient? host sends respective parts of the data to each node nodes read their part of the data from database How efficient are broadcasts (are they used at all)?

15 Resampling for SNP-Data Outlook & Questions 15/ 16 Last but not least... What about the infrastructure? data storage, data bases, maintenance, limits? sharing of data / results (community) R-package used for parallel computing? installation of own R-packages (+Bioconductor) problems with parallel R-computing and heterogeneous hardware help and support? test runs? speedups for certain tasks?

16 Resampling for SNP-Data Outlook & Questions 16/ 16 Thank you for your attention.