Institut Français de Bioinformatique, Un Cloud pour les Sciences du Vivant

Transcription

1 Institut Français de Bioinformatique, Un Cloud pour les Sciences du Vivant Christophe Blanchet! Institut Français de Bioinformatique - IFB French Institute of Bioinformatics - ELIXIR French Node CNRS UMS Gif-sur-Yvette - FRANCE

2 Sequencing data source: source: Complete genome sequencing become a lab commodity with NGS (cheap and efficient)

3 And other experimental data FR - EU

4 EMBL-EBI data resources growth source: EMBL-EBI Annual Report 2013

5 Plateformes Expérimentales en Biologie Plateformes nationales (GIS IBISA) Nb Imagerie cellulaire 19 Génomique, Transcriptomique 16 Protéomique 13 Biologie structurale, biophysique 11 NGS BI C IMG Biological platform (Genomics, IMaGing, PROteomics...) Bioinformatics center Cloud resources Scientists PRO C NGS BI NGS PRO BI C Localisation des plateformes NGS PRO BI IMG PRO NGS PRO C IMG BI C C Source: omicsmaps.com Des sites intermédiaires permettent de répartir la charge en terme de stockage et de puissance de calcul tout en assurant une meilleure proximité avec les scientifiques

6 Infrastructures in Biology Lot of bioinformatics tools and services to treat and vizualize the biological data

7 IFB mission IFB: national research infrastructure of service in Bioinformatics! Mission: to make available core bioinformatics resources to the national life science research community.! by providing support for biology programs! projects bolstering! users training! by setting up an IT infrastructure devoted to the management and analysis of biological data! material resources : CPUs, disks, etc.! availability of biology data collections! deployment of bioinformatics tools (Cloud)! by acting as a middleman between the life science community needs and the bioinformatics/computer science research community! by being the ELIXIR French node

8 IFB structure IFB consists of :! A network of 6 regional centers (> 20 PFs)! about 110 FTE permanent staff + 70 FTC staff! A national node : IFB-core! plan of 10 FTE permanent staff (currently 3) + a few FTC staff

9 IFB-core: IFB s national hub IFB-core consists of two teams: The "Communication, Training, Exploitation" unit in charge of the web site and training program based on e-learning The "Infrastructure" unit in charge of the IFB national IT infrastructure

10 IFB-core tasks to provide a technical and administrative support to IFB to implement IFB scientific policy and facilitate the dissemination of actions to ensure an effective coordination between the PFs to serve as an interface by providing a unique entry point to IFB partners (supervisory authorities, the life science community, European and national bioinformatics communities) to set up and manage the IFB s national IT infrastructure to facilitate access to this IT infrastructure by deploying an academic Cloud

11 IFB e-infrastructure Support : help members to deploy and use their tools e-infrastructure: hardware, biology data collections, bioinformatics tools Academic cloud for life science a core ressource IFB-core hosted at CNRS IDRIS SC center (Paris) + regional resources 6 regional bioinformatics centers with 2 clouds 11,000 cores - +6 PB but +20 bioinformatics platforms Create a federation of clouds for life sciences Technical organization GRISBI: a national technical working group (all platforms) Participation to ELIXIR task forces RENABI-GO RENABI-SO APLIBIO RENABI-NE IFB-core PRABI RENABI-GS Cloud Ressources Location # Compute Cores # TB Storage # TB RAM Max VM size Technology IFB-core CNRS-IDRIS, Paris c 256GB StratusLab IFB-core 2015 CNRS-IDRIS, Paris 3, c 1TB StratusLab IFB-core 2016 CNRS-IDRIS, Paris 10,000 2,000-96c 2TB StratusLab Genocloud IFB-GO, Rennes ONE

12 Cloud? public community private SaaS Deploy. hybrid PaaS IaaS Ack.: C. Loomis

13 A cloud driven through a web dashboard

14 Ready-to-use bioinformatics cloud appliances Bioinformatics appliances are usual virtual machines small : few GB, easy to convert in most virtualization formats Installed and preconfigured with bioinformatics tools e.g. BLAST, Clustalw, ARIA, MEME, HMMer, TopHat, BWA, Samtools, etc. Referenced in a marketplace devoted to bioinformatics tools BLAST FastA OMSSA ClustalW2 SSearch PeptideShaker ARIA BWA X!tandem HMMer TopHat samtools Galaxy Clustal Muscle fastqc Omega Create new cloud services R Linux system Bioinformatics Marketplace Structures Sequences Virtual Machines Proteomics + Galaxy...

15 Current bioinformatics IFB Scientific apps CLI Virtual desktop Web Galaxy MODAL Proteomics Galaxy Galaxy AVIESAN 2013 RSAT PhyML RSAT mini biocompute Aria Node Utilities biodata BioMaj BlobSeer biodata NFS Cassandra Data mgmt biohadoop CentOS Ubuntu Base OS

16 Run bioinformatics appliances Bioinformatics marketplace both a virtual machines repository Store life science VMs and a catalogue Help users to select the appropriate VM for their analysis Bioinformatics Marketplace BI Structures Sequences Proteomics B A data public data UNIPROT EMBL Genomes PDB PROSITE Move cloud virtual machines Analyze data Galaxy tools VM: BLAST, ClustalW2, etc.... (2) IDB Cloud (3) Select tools Scientists can filter (1) the appliances through a Web interface to identify and launch (2) the appropriate ones. (1) Use tools (3) Scientists have access to their own cloud resources through web portal, remote virtual desktop or SSH. Filter images with metadata related to bioinformatics attribute <bio:tool> in VM manifests scientists can select the appropriate appliance according to the tools required for their analyses e.g. the BLAST tool Deploy on several clouds

17 Storage for biological data CLI (scp, sftp), GUI (Cyberduck, Transmit, Filezilla, ) sftp/http/s3 Upload your data Public Data sources Genomes EMBL PDB UNIPROT PROSITE shared (NFS ro) BLAST, Clustal, etc. PaaS IaaS launch jobs ssh Shared FS Master & Storage VM ARIA Workers VM CNS Identity Mgmt j. doe e. martin you chb virtual disks Portal Bioinformatics Cloud cg User data sftp/http/s3 Get your results CLI (scp, sftp), GUI (Cyberduck, Transmit, Filezilla, )

18 Monitor your usage

19 Moving VMs vs Data NGS IMG PRO NGS Biological platform (Genomics, IMaGing, PROteomics...) BI C Bioinformatics center Cloud resources Scientists C BI NGS data PRO VM BI VM C VMs PRO IFB life sciences marketplace & VMs repository NGS data VM PRO BI C IMG PRO data IMG BI C C

20 Case 1: Standard Bioinformatics node Use your own instance(s) With pre-installed standard appliance Biocompute bioinformatics tools BLAST, FastA, SSearch,HMM,... ClustalW2, Clustal-Omega, Muscle,.. Bowtie(2), BWA, samtools,... MEME, R, etc. Connected to public reference data Uniprot, EMBL, genomes, PDB, etc. Automaticaly shared to the VMs Cluster mode turn several instances in a single virtual cluster shared file system batch scheduling

21 Case 2: Cloud Galaxy portal Web interface portal Galaxy is widely used in the community analyse NGS data (mainly but not only) connected to community knowledge: data and indexes, tools, workflows Preserve workflows and results (cloud virtual disk) Help the integration of monthly updates and new tools Cloud permit different appliances to be built from the same base: generic with common tools for NGS specific for a set of tools, example of MODAL (MOdels for Data Analysis and Learning) for training: create a special appliance with dedicated datasets, tools or workflows (French AVIESAN school 2013) domain specific appliances: RNAseq, CHIPseq, etc. (planned)

22 Galaxy IFB cloud manual installation Linux system Created by interactive installation Main with standard tools Devoted For a specific event: training, demo, For tools: e.g. MODAL Galaxy Galaxy Galaxy AVIESAN 2013 MODAL Version 1.0 Version 2.0 N

23 Case 3: A specialized software suite for the analysis of noncoding sequences motif discovery in promotors of co-expressed genes CHIPseq analysis evolutionary conserved motifs (phylogenetics footprints) Contact: J. van Helden (TGAC) Used for ECCB 14 tutorial T01 RSAT offers a series of tools dedicated to the detection of regulatory signals in noncoding sequences input a list of genes of interest you retrieve the upstream sequences over a desired distance, discover putative regulatory signals, search the matching positions for these signals in your original dataset or in whole genomes, display the results graphically in the form of a feature map.

24 Case 4: Proteomics virtual desktop Motivation Collaboration with a mass spectroscopy platform Running out of space on their local resources Protein identification tools Mass experimental data Reference databases : nr, Swiss-Prot Reference screening tools: OMSSA, X!Tandem User interface Remote Virtual Desktop (NX) Reference GUIs SearchGUI PeptidShaker source: PeptideShaker site

25 IFB - an academic cloud for life sciences IFB s cloud simplify access to biological data and tools integrate tools and pipelines in turnkey cloud appliances is tightly connected to existing bioinformatics resources, e.g. public reference data sources 16 bioinformatics appliances: standard compute nodes, proteomics virtual desktop, Galaxy portal, structural biology +50 users from all IFB regional centers Bioinformatics appliances registry reference images related to life science help users to select the appropriate VM for their analysis BI tools BLAST FastA OMSSA ClustalW2 SSearch PeptideShaker ARIA BWA X!tandem HMMer TopHat samtools Galaxy Clustal Muscle fastqc Omega Create new cloud services Virtual Machines R Linux system Bioinformatics Marketplace Bioinformatics Marketplace Structures Galaxy Sequences Proteomics Proteomics B A data public data UNIPROT EMBL Genomes PDB PROSITE Move cloud virtual machines Analyze data tools VM: BLAST, ClustalW2, etc Galaxy (2) IDB Cloud... (3) Selec Scientists can the appliances Web interface t and launch appropriate ones (1) Use Scientists have access cloud resources throug remote virtual desktop o

26 Perspectives Create more bioinformatics appliances by the experts of the domains publish them in the IFB marketplace to make them available to the scientists! IFB established priorities for 5 scientific domains Microbial Bioinformatics Evolutionary bioinformatics Plant bioinformatics Structural Biology NGS data processing! and technical pilots Interoperability of appliances on different cloud infrastructures Registry of distributed multi-cloud datasets Live remote cloud processing of sequencing data

27 Questions? Acknowledgments! Clément Gauthey (CNRS IDRIS, form. IDB-IBCP) Developers of tools that integrated them as an IFB cloud appliance: Samuel Blanck (Inria Lille), Jacques van Helden (TAGC), You?! StratusLab members! IFB s funding by French program PIA INBS 2012