Une e-infrastructure nationale en bioinformatique

Transcription

1 Une e-infrastructure nationale en bioinformatique Christophe BLANCHET Institut Français de Bioinformatique - IFB French Institute of Bioinformatics - ELIXIR-FR CNRS UMS Gif-sur-Yvette - FRANCE JDEV juillet 2015, Bordeaux

2 History Since 2004, ReNaBi is the National Network of Bioinformatics platforms with an IBiSA label (Infrastructures in Biology, Health and Agronomy) In 2010, call of proposals Infrastructures in Biology and Health from the Investments for the Future initiative. Project ReNaBi-IFB accepted in 2012 and endowed with 20m Other national infrastructures (NIs) France Génomique : sequencing and genotyping NI Profi : proteomics NI Frisbi : structural biology NI etc. (17 NIs all together) + 5 IHUs (Instituts Hospitaliers Universitaires) + 1 IRT (Institut de Recherche Technologique) 2

3 IFB - Ins0tut Français de Bioinforma0que IFB, the French distributed infrastructure for life-science information Mission : to make available core bioinformatics resources to the national/international life science research community. To provide support for national biology programs To provide an IT infrastructure devoted to management and analysis of biological data To act as a middleman between the life science community and the bioinformatics/ computer science research community CNRS UMS3601. Avenue de la Terrasse, Bât Gif-sur-Yvette ELIXIR French Node The European distributed infrastructure for lifescience information To optimize the interactions and coordination between the national level and ELIXIR and other ESFRI infrastructures in biomedical and environmental field, To promote consistency and complementarities between the components offered by the ELIXIR French node and those of other European nodes 3

4 Experimental data in life sciences (FR) French national platforms (GIS IBISA) Nb Cellular imaging 19 Genomic, Transcriptomic 16 Proteomic 13 French NGS platforms Structural biology, biophysic 11 NGS C BI PRO Source: omicsmaps.com NGS IMG PRO NGS BI C Biological platform (Genomics, IMaGing, PROteomics...) BI C Bioinformatics center Cloud resources Scientists BI IMG PRO NGS BI C PRO IMG C C Un déluge de donnée. Blanchet C. et Collin O., 2011, Biofutur, 323: PRO Regional centers distribute the load in terms of computing and storage, and provide better interactions with scientists Des sites intermédiaires permettent de répartir la charge en terme de stockage et de puissance de calcul tout en assurant une meilleure proximité avec les scientifiques 4

5 A lot of public reference databases source: EMBL-EBI Annual Report

6 A lot of bioinforma0cs tools tools BLAST FastA OMSSA ClustalW2 SSearch PeptideShaker ARIA BWA X!tandem HMMer TopHat samtools Galaxy Clustal Muscle fastqc Omega R ABYSS ARIA 2.3 Bioconductor 2.11 biomaj BLAST Blat 35 Bowtie Bowtie beta7 BWA BWA CAP3 CD-HIT Clustal Omega CLUSTALW 2.1 Cufflinks Cutadapt E-SURGE Exonerate express FastA 3.6 FastQC Galaxy portal GATK HMMer 3.0 ImageJ 1.48 khmer 1.1 M-SURGE MEME 4.7 MMSEQ a Mobyle MODAL MultAlin MUSCLE neo4j Oases OMSSA PeptideShaker phyml 3.1 PREDATOR proline python 2.7 R 2.13 R R R-studio Ray 1.3 RSAT samtools Samtools 1.1 SearchGUI SeqClean Shiny Stacks STAR 2.4.0f1 SuMo v1 TGICL TopHat trim_galore Trinity U-CARE VCFtools Velvet X!tandem XPLOR-NIH

7 Many interfaces 7

8 Use case : deploy a simple bioinforma0cs applica0on Use case AuthN/Z Cloud web dashboard VMs lifecycle log in + data(+) deploy Pipeline web interface isolated network VM user s data User AuthN/Z +encrypt. refdata Cloud SW IFB web dashboard StrastusLab Slispstream TRESOR Docker SAML/ XCAML Cluster OpenNaaS Hadoop ( ) HW Secure User s LAN WAN academic + public DC A refdata 8

9 Use case: deploy a complex bioinforma0cs applica0on Use case AuthN/Z User Cloud web dashboard VMs/disks lifecycle AuthN/Z deploy data (scp/sftp) log in I/O viz viz isolated network shared data worker worker worker worker worker worker worker worker worker worker Bioinformatics web interface Cloud SW IFB web dashboard StrastusLab Slispstream SAML/ XCAML TRESOR OpenNaaS ( ) refdata(+) HW User s LAN WAN academic + public Isolated network DC A DC B geno mes Sync geno mes refdata(+) 9

10 Deploy and operate IFB s e- infrastructure as a cloud 10

11 IFB e- Infrastructure Mission : to provide core bioinformatics resources to the life science research community. To set up a French IT infrastructure devoted to management and analysis of biological data To collaborate with international infrastructure (ELIXIR) Goal : to help scientists and engineers to deploy and use their tools e-infrastructure: provide hardware, data collections and bioinformatics tools Current resources A national hub: IFB-core hosted at CNRS IDRIS SC center A network of regional centers: 32 bioinformatics platforms - 15,000 cores - 5 PB Create a federation of clouds for life sciences 11

12 IFB s CloudS? IFB-GO APLIBIO IFB-NE Running clouds IFB-core GenOuest IFB-core PRABI PoC & experiments URGI BiLille/Univ.Lille BISTRO/IPHC IFB-SO IFB-GS Next Lyon?? 12

13 IFB- core s cloud IFB-core # Compute Cores # TB Storage # TB RAM Max VM size Technology Location Pilot c 256GB StratusLab CNRS-IDRIS, Paris 2016-S1 3, ?144c 3TB? StratusLab ,000 2,000 -?? StratusLab CNRS-IDRIS, Paris CNRS-IDRIS, Paris NGS, imaging, statistics, PaaS IaaS launch jobs Scientists RENATER 10giga SaaS Frontend Master Virtualization Layer Shared FS Workers Web portal Pdisk storage iscsi 10giga eth IDRIS CNRS SC-center 10giga eth Cloud Hypervisors - std nodes: 32c 128GB - bigmem nodes: 40c 256GB 13

14 Create bioinforma0cs appliances tools BLAST FastA OMSSA ClustalW2 SSearch PeptideShaker ARIA BWA X!tandem HMMer TopHat samtools Galaxy Clustal Muscle fastqc Omega Create new cloud services Virtual Machines R + Linux system Bioinformatics Marketplace Appliance? predefined virtual machine Ready to run Description Title Contact (and maintainer!) Description (w. controlled voc.) + Structures Sequences Proteomics Galaxy... 14

15 Docking bioinforma0cs tools IFB docker hub Registry of images Bioinformatics images Docker Image size (MB) abyss pull push IFB s Cloud blast blast+ bowtie bwa clustalw2 cufflinks docker virtual machine docker virtual machine fasta36 fastqc gor4 Developer User hmm meme mmseq cufflinks

16 Move VMs rather than data NGS C BI NGS data PRO VM BI VM C PRO VMs IFB s marketplace & VMs repository for life sciences NGS data VM PRO BI C IMG PRO data IMG BI NGS BI IMG PRO Biological platform (Genomics, IMaGing, PROteomics...) Bioinformatics centre C C C Cloud resources Researchers 16

17 IFB s bioinforma0cs cloud services 17

18 A cloud driven through a web dashboard 18

19 Storage for biological data CLI (scp, sftp), GUI (Cyberduck, Transmit, Filezilla, ) sftp/http Upload your data Public Data sources Genomes EMBL PDB UNIPROT PROSITE shared (NFS ro) BLAST, Clustal, etc. PaaS IaaS launch jobs ssh Shared FS Master & Storage VM ARIA Workers VM CNS Identity Mgmt j. doe e. martin you chb virtual disks Portal Bioinformatics Cloud cg User data sftp/http Get your results CLI (scp, sftp), GUI (Cyberduck, Transmit, Filezilla, ) 19

20 IFB s bioinforma0cs appliances Remote desktop Web Proteomics Galaxy Scientific apps Galaxy Galaxy MODAL AVIESAN 2013 RSAT Galaxy RADseq PhyML R statistics CLI biocompute Aria Node Imaging Eco Pop Utilities biodata BioMaj BlobSeer biodata NFS Cassandra Neo4j Data mgmt biohadoop Docker CentOS Ubuntu Base OS 20

21 Browse the marketplace and run an App! Proteomics Sequences Galaxy Structures?... IFB s bioinformatics marketplace! 21

22 RAINBio : Registry of bioinforma0cs tools and VMs Prototype Query: topic? tool? VM? Cloud Marketplace e.g. IFB VMs Services Registry e.g. ELIXIR Tools Life science researcher RAINBio Graph DB (Neo4J) 22

23 App Biocompute Standard bioinformatics node With pre-installed standard bioinformatics tools BLAST, FastA, SSearch,HMM,... Bowtie(2), BWA, samtools,... MEME, R, etc. ClustalW2, Clustal-Omega, Muscle,.. Connected to public reference datasets Uniprot, EMBL, genomes, PDB, etc. Automatically shared with the VMs Cluster mode turn several instances in a single virtual cluster shared file system batch scheduling 23

24 App R Sta0s0cal Compu0ng R software environment for statistical computing and graphics include common bioinformatics module RStudio IDE Biobase, BiocGenerics, BiocInstaller, GenomeInfoDb integrated development environment (IDE) for R features: console, syntax-highlighting editor Shiny web framework powerful web framework for building web applications using R. without requiring HTML, CSS, or JavaScript knowledge. Contact: Stéphane Delmotte (PRABI-LBBE) 24

25 Motivation App Proteomics desktop Collaboration with a mass spectroscopy platform Running out of space on their local resources Protein identification tools Mass experimental data Reference databases : nr, Swiss-Prot Reference screening tools: OMSSA, X! Tandem User interface Remote Virtual Desktop (NX) Reference GUI PeptidShaker 25

26 And others RSAT Galaxy Databank FastAMR splits the databank into subsets and puts them in the DFS along with the sequences file FastAMR subset #01 FastA #01 subset #02 Mappers FastA # Each mapper send the score and sequences to reducers Reducers Results score sequence score sequence... Users run the FastAMR script with its sequences and the databank User's Sequences Each mapper runs a FastA program on a part of the databank Reducers copy the best scores of the whole experiment in the DFS Ecology of populations Hadoop 26

27 Conclusion 20 bioinformatics appliances already available + 10 in progress IFB supports different domain-specific developments Microbial Bioinformatics, Evolutionary bioinformatics, Plant bioinformatics, Structural Biology, NGS data processing Scientific production users (July 2015) opened to members of IFB (standard allocated resources) opened to partners, academic and industry, infrastructures and projects: e.g. BioDataCloud, ProFi, MetaboHub, extra resources allocation according to scientific and financial criteria Training - 70 users French tutorials Cloud pour la Biologie tutorial at ECCB 14 about Analysis of Cis-Regulatory Motifs from High- Throughput Sequence Sets Bioinformatics Master in Marseille (2014), in Rouen (2015) 27

28 Perspec0ves Create more bioinformatics appliances available to the community By the experts of the different life sciences domains Appliances in progress: BioDataCloud-RNAseq, ProFi, SymBioWatch, Clinical NGS for cancerology (2x), REPET, TriAnnot, Galaxy RAD-seq, Bacterial genomics, imetamos, BioDataCloud-Genomes-Browser Call in 2015-Q1 for proposals to support technological developments and integration of data and tools Pilots Multi-cloud deployment of complex applications Secure cloud for human biomedical data analysis Live remote cloud processing of sequencing data Data registry of multi-cloud datasets Docker hub for life sciences 28

29 Ques0ons? Acknowledgments IFB members IFB hub: Patricia, Marie, Jean-François, Mohamed, Quentin we are hiring! Groupe de réflexion sur les InfraStructures BioInformatiques IFB-GRISBI (co-chair with Olivier Collin) Appliances developers: Samuel Blanck (Inria Lille), Jacques van Helden (TAGC), Stéphane Delmotte (PRABI-Doua), Bruno Spataro (PRABI-Doua) CNRS IDRIS: R. Medeiros, C. Gauthey and staff StratusLab members IFB is funded by French program PIA INBS 2012 EU H2020 EGI-Engage (654142) and CYCLONE (644925) projects 29