R, paralelización, datos masivos y aplicaciones web: ejemplos del uso de R en bioinformática

Transcription

1 R, C, R, paralelización, datos masivos y aplicaciones : ejemplos del uso de R en bioinformática Ramón Díaz-Uriarte Dept. Bioquímica Universidad Autónoma de Madrid Madrid, Spain rdiaz02@gmail.com III Jornadas de Usuarios de R 17-Noviembre-2011 (1 : 58)

2 License copyright R, C, This work is Copyright, c, 2011, Ramón Díaz-Uriarte, is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this license, visit by-nc-sa/3.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. ***************************** Please, respect the copyright. This material is provided freely, if you use it, I only ask that you use it according to the (very permissive) terms of the license: attribution, non-comercial use, a share alike license. If you have any doubts, ask me. (2 : 58)

3 Outline R, C, R, C, (3 : 58)

4 Biological context Computational context R, C, Biological context Computational context R, C, (4 : 58)

5 Chromosomes Biological context Computational context R, C, From the Wikipedia; original source Glossary/Illustration/karyotype.shtml (5 : 58)

6 DNA protein Biological context Computational context R, C, (From O. Rueda s PhD Thesis) (6 : 58)

7 Data from a microarray experiment Biological context Computational context R, C, Slide from Gema Moreno Bueno, Department of Biochemistry, UAM (7 : 58)

8 More microarray data Biological context Computational context R, C, Modified from students/peter_cock/r/heatmap/scaled_color_key.png (8 : 58)

9 DNA protein Biological context Computational context R, C, (From O. Rueda s PhD Thesis) (9 : 58)

10 Log 2 (Ratio) R en acgh Barrett et al., 2004 Biological context Computational context Calling gains losses: hypothesis testing R, C, Hupe & Barillot, 2005 Inferring number of copy gains/losses: estimation Olshen, 2005 Arrays: a dot is a DNA fragment. Each array a sample. Each array all chromosomes. (For analysis, location in chromosome matters) Chromosome (10 : 58)

11 Data, data, data (in Gigabytes) Biological context Computational context R, C, Expression arrays (mrna) > 40,000 probes Copy number with acgh > 400,000 common; some > 4 x (11 : 58)

12 Biological context Computational context R, C, Multicores computing clusters Increases in CPU speed slowed down (< 20% per year since 2002). Increase in the number of cores : 2, 4, 8. Next 10 years? Inexpensive computing clusters with off-the-shelf components. Must design our programs from the start: parallel programming Image from (12 : 58)

13 R, C, R, C, (13 : 58)

14 Stalone Statistical Computing in Bioinformatics R, C, Develop statistical methods Implement for statisticians bioinformaticians Implement existing approaches Implement for wet lab users - Parallel Computing - Fault tolerance Web apps: - User friendly - No installation Increased speed (40x - 60x) - Statistical rigour - Best practices (14 : 58)

15 R R, C, Code available for many procedures (but a few years ago none parallelized!) Many computations embarrassingly parallelizable: bootstrapping cross-validation arrays (or samples) arrays by chromosomes parallel chains in MCMC Figure production can be parallelized (15 : 58)

16 R R, C, (Implement missing functionality: R/C) MPI: R packages Rmpi, papply, snow, snowfall Load balanced Wrappers over mid level functions in package: ease updating Parallelize: Bootstrap samples/cross-val. runs. arrays arrays by chromosomes (or a combination of both) (16 : 58)

17 Is it worth it? R, C, Are speed improvements really worth the effort? Over what range of problems do see improvements? With what hardware can we see improvements? (17 : 58)

18 What do we gain? HMM 20,000 genes GLAD CBS BioHMM Web applications Large 5000 data sets 2500 User wall time (seconds) User wall time (seconds) R, C, Sequential compression 1000 on Parallelized Number of arrays (samples) (18 : 58)

19 What do we gain? R, C, Are speed improvements really worth the effort? Your effort: R CMD INSTALL ADaCGH2. Over what range of problems do see improvements? 10 to 10 3 arrays/samples; 10 4 to 10 6 spots/genes. With what hardware can we see improvements? 2 cores to 120 cores. Smaller clusters: more cost effective Single node/multi-core: less communication overhead (19 : 58)

20 Where is this running? R, C, varselrf (CRAN) ADaCGH2 (BioConductor) SignS (launchpad: (20 : 58)

21 Web apps: how R, C, Web apps: how R, C, (21 : 58)

22 Applications for wet lab researchers Web apps: how R, C, Analyze data in a reasonably short time. User friendly access to methods that are statistically rigorous. (22 : 58)

23 Web apps: how R, C, Web-based applications User-friendly interface. No hardware/software hassles for end users. Parallelization is transparent. Method selection can be partially transferred (to us). Short user wall time: use (hardware/software) resources rarely available to individual biomedical researchers Just type in a URL: Image modified from (23 : 58)

24 Sometimes collaborations feel like... Web apps: how R, C, (From (24 : 58)

25 Parallelization in applications Statistical Computing in Bioinformatics Web apps: how Develop statistical methods Implement existing approaches R, C, Implement for statisticians bioinformaticians - Parallel Computing - Fault tolerance Implement for wet lab users Web apps: - User friendly - No installation Increased speed (40x - 60x) - Statistical rigour - Best practices (25 : 58)

26 Main -based applications Dealing with raw data Remove artifacts from microarrays - Missing data - Replicate spots Web apps: how DNMAD prep R, C, Segment acgh Molecular signatures survival data Differentially expressed genes Statistical analysis (sensu stricto) Select genes for classification WaviCGH ADaCGH SignS Pomelo_II Tnasas GeneSrF Annotation Interpretation Interpret results IDClight PaLS (26 : 58)

27 Web apps: how R, C, User wall time (seconds) What do we gain? GLAD CBS genes, 40 arrays (27 : 58) CGHseg HMM Number of simultaneous users

28 How it works: some key ideas Web apps: how R, C, Each run Parallelization (transparent for users) Fault-tolerance (network problems, machine crashes, bugs) Check-pointing Periodic tasks (keep system running 24h, 365 d) Automatic monitorization Automated testing suite (28 : 58)

29 What happens Web apps: how R, C, User Head node (LVS): Send request to one of the servers. - Setting up LAM/MPI - Starting R - Fault tolerance - Checking termination of R - Checking run errors - Formatting output CGI: data checking, file upload Execution: Python program R program Autorefreshing HTML until final results Sequential Parallelized (29 : 58)

30 What happens: details User Web apps: how Monitor R execution Maintain R process counters CGI Read data Head node (LVS) Server 2 (Master) Apache Create MPI universe Launch R, Rmpi R, C, Is R done? Yes No Return autorefreshing page (slave) (slave) (slave) Yes Server n Server 1 Server 3 Continue R execution till end Rmpi started OK? Not after K attempts No Stop execution Halt MPI universe Return error Halt MPI universe Produce return results pages (30 : 58)

31 MPI details Sleep No Can we run? (Count other lam daemons) NFS shared storage Verify servers (modify LAM defs) Web apps: how Yes Boot (new) LAM/MPI R, C, Start R: continue from last checkpoint Segmentation Figures (over subjects chrom.). Sleep No No Yes MPI universe: Servers 1... n Run out of time? Are we done? R crashed (bugs)? Rmpi crashed? LAM/MPI/nodes crashed? Yes Halt MPI universe Produce return results pages NFS shared temporary storage (31 : 58)

32 Where is this running? Web apps: how R, C, (32 : 58)

33 R, C, R, C, (33 : 58)

34 Log 2 (Ratio) R en acgh Barrett et al., 2004 Calling gains losses: hypothesis testing R, C, Hupe & Barillot, 2005 Inferring number of copy gains/losses: estimation Olshen, 2005 Arrays: a dot is a DNA fragment. Each array a sample. Each array all chromosomes. (For analysis, location in chromosome matters) Chromosome (34 : 58)

35 R, C, Millions of spots Hundreds or thouss of subjects. No need to hold everything in RAM at once. Package ff: memory-efficient storage of large data on disk fast access functions. Combined with: shared storage (35 : 58)

36 ff R, C, ff stores the object on disk. Read that object from various R processes. Different R processes can write in different ff objects (36 : 58)

37 ff (I) Common ff object R, C, write read only R 1 R 2 R n ff 1 ff 2 ff n (37 : 58)

38 ff (I) Common ff object R, C, write read only R 1 R 2 R n ff 1 ff 2 ff n R master (37 : 58)

39 ff (I) Common ff object R, C, write read only R 1 R 2 R n ff 1 ff 2 ff n R master ff all (37 : 58)

40 ff (II) write Data R master read only R, C, ff in (38 : 58)

41 ff (II) write Data R master read only (multicore) R, C, ff in (38 : 58)

42 ff (II) write Data R master read only R, C, ff in R 1 R 2 R n ff 1 ff 2 ff n (38 : 58)

43 ff (II) write Data R master R, C, read only i 1 ff in i 2 i n R 1 R 2 R n (38 : 58)

44 ff (II) write Data R master R, C, read only i 1 ff in i 2 i n R 1 R 2 R n (38 : 58)

45 ff (II) write Data R master read only R, C, ff in R 1 R 2 R n Fig.1 Fig.2 Fig.n ff 1 ff 2 ff n (38 : 58)

46 ff (II) write Data R master read only R, C, ff in R 1 R 2 R n Fig.1 Fig.2 Fig.n ff 1 ff 2 ff n (38 : 58)

47 ff (II) ff out Results write Data R master read only R, C, ff in R 1 R 2 R n Fig.1 Fig.2 Fig.n ff 1 ff 2 ff n (38 : 58)

48 Where is this running? R, C, ADaCGH2 (BioConductor package) Web-based application (39 : 58)

49 R, C, R, C, (40 : 58)

50 Log 2 (Ratio) R en acgh Barrett et al., 2004 Calling gains losses: hypothesis testing R, C, Hupe & Barillot, 2005 Inferring number of copy gains/losses: estimation Olshen, 2005 Arrays: a dot is a DNA fragment. Each array a sample. Each array all chromosomes. (For analysis, location in chromosome matters) Chromosome (41 : 58)

51 Store access (large) pre-computed results R, C, HMM for acgh data with Reversible Jump: Viterbi Common regions: count on the Viterbi paths. Fitting HMM/common regions: distinct operations. C: number-crunching. R: wrapper figures/tables. C: creates large amounts of data. In package RJaCGH (CRAN). (42 : 58)

52 R Fit HMM C (HMM) Store Viterbi as gzipped file R, C, (43 : 58)

53 R, C, R Fit HMM C (HMM) return filenames Store Viterbi as gzipped file (43 : 58)

54 R Fit HMM C (HMM) return filenames Store Viterbi as gzipped file R, C, Find common regions R pass filenames C (common regions) (43 : 58)

55 R Fit HMM C (HMM) return filenames Store Viterbi as gzipped file R, C, Find common regions R pass filenames C (common regions) return results Read Viterbi data Figures, tables (43 : 58)

56 R, C, R, C, (44 : 58)

57 Web-based: A few things we ve learned R, C, Configuration sucks (if you need to modify > 1 file) Too many languages Adding test cases to the testing suites:, R Documentation: in the, pages, L A T E X... Too much R to catch errors User interfaces: who designs them? (45 : 58)

58 Fault tolerance communication Manual check for errors (R ain t Erlang) Too much network traffic NFS shared storage Verify servers (modify LAM defs) Boot (new) LAM/MPI Start R: continue from last checkpoint Sleep Yes R, C, No No MPI universe: Servers 1... n Run out of time? Are we done? R crashed (coding errors)? Rmpi crashed? LAM/MPI crashed? (includes node crashes) Yes Halt MPI universe Produce return results pages NFS shared temporary storage (46 : 58)

59 Solutions? R, C, Literate programming org-mode Alternatives to MPI /or use Erlang... Keep things as they are (only a few painful events a year) (47 : 58)

60 Single machine applications R, C, Run applications in a single, multicore (e.g., 12) machine Just verify if the machine is up (48 : 58)

61 Rethinking -based applications Users can get into trouble. R, C, (49 : 58)

62 Rethinking -based applications R, C, Users can get into trouble. Sure, but we can do a good job... Provide state-of-the art statistical computational approaches (R ;-) Pedagogical examples pipelines Minimize the chance of users getting into trouble (49 : 58)

63 Rethinking -based applications R, C, Users can get into trouble. Sure, but we can do a good job... Provide state-of-the art statistical computational approaches (R ;-) Pedagogical examples pipelines Minimize the chance of users getting into trouble Web-based applications are here to stay (49 : 58)

64 ... so... R, C, Forget about them: just write R (plus C, etc) Go for it R will do its job (which is only part of the job) HPC available No problem with large data sets But other tools work necessary (50 : 58)

65 Regardless of -based applications... R, C, Parallel computing can be used routinely (library(parallel) in R ) with ff +. (51 : 58)

66 Acknowledgements R, C, O. M. Rueda, A. Alibés, A. Cañada, E. R. Morrissey, M. L. Neves, D. Rico. Funding: Fundación de Investigación Médica Mutua Madrileña, Project TIC C02-02 of the Spanish MEC BIO of the Spanish MICINN. Ramón y Cajal Programme of the Spanish Ministry of Education Science. CNIO (Spanish National Cancer Research Center). The R users developers for a vibrant statistical computing community amazing platform. The organizing scientific committees of the III Jornadas de Usuarios de R. (52 : 58)

67 Trying to fit it all Statistical Computing in Bioinformatics R, C, Implement existing approaches Implement for statisticians bioinformaticians - Parallel Computing - Fault tolerance Develop statistical methods Implement for wet lab users Web apps: - User friendly - No installation Information integration Increased speed (40x - 60x) - Statistical rigour - Best practices Answer biological question (53 : 58)

68 What do we gain? Effect of number of arrays (number of genes = 7399) Effect of number of genes (number of arrays = 160) Fold increase in speed (relative to 1 CPU) R, C, compression 5.0 on future, 2.0 et al Fold increase in speed (relative to 1 CPU) Original Sequential New sequential (1 CPU) Parall. 2 CPUs Parall. 10 CPUs Parall. 20 CPUs Parall. 60 CPUs Number of arrays (samples) (54 : 58) Number of genes

69 What do we gain? R en Breast data set (78 arrays x 4751 genes) DLBCL data set (160 arrays x 7399 genes) 2500 Large data 2000sets User wall time (seconds) R, C, User wall time (seconds) Number of simultaneous users Number of simultaneous users (55 : 58)

70 Tools... Org mode Literate programming R, Python, C/C++ Interactive debugging Web app. frameworks Bazaar-NG Commit hooks R, C, Admin. ConfigFiles WebHelp FunkLoad: - Whole system testing - Regression testing - Stress testing ''Continuous Integration'' InstallScript Distributed computing Parallel. (MPI, others) Grid computing Web services (56 : 58)

71 R, C, Too many languages Impedance mismatch problem: Building Web-based applications requires the mastering of a number of languages/technologies (e.g. HTML, CSS, CGI, ASP, PHP, XML, etc..). Such languages technologies were created to address different aspects on a by-need evolutionary manner. The result is a plethora of tools that are fitted together in an ad hoc fashion. El-Ansary, Grolaux, Van Roy, Rafea (2005) Overcoming the Multiplicity of Languages Technologies for Web-Based Development Using a Multi-paradigm Approach. R C HTML Python: CGI, data entry, display Python ( others): control monitor MPI Javascript: AJAX figures (57 : 58)

72 Other solutions? R, C, Too many languages Use languages designed to overcome this problem: Hop, Links, QHTML. Fault tolerance too much traffic Alternatives to MPI? Linda tuple spaces (also between-language funct.) Roll-our-own based on Rserve Have Erlang control R processes? (58 : 58)