Le cloud IFB et son instance Galaxy

Transcription

1 Le cloud IFB et son instance Galaxy Christophe BLANCHET Institut Français de Bioinformatique - IFB French Institute of Bioinformatics - ELIXIR-FR CNRS UMS Gif-sur-Yvette - FRANCE Ecole Bioinformatique Aviesan 28 Septembre 2015, Roscoff

2 Experimental data in life sciences (FR) French national platforms (GIS IBISA) Nb Cellular imaging 19 Genomic, Transcriptomic 16 Proteomic 13 French NGS platforms Structural biology, biophysic 11 NGS C BI PRO Source: omicsmaps.com NGS IMG PRO NGS BI C Biological platform (Genomics, IMaGing, PROteomics...) BI C Bioinformatics center Cloud resources Scientists BI IMG PRO NGS BI C PRO IMG C C Un déluge de donnée. Blanchet C. et Collin O., 2011, Biofutur, 323: PRO Regional centers distribute the load in terms of computing and storage, and provide better interactions with scientists Des sites intermédiaires permettent de répartir la charge en terme de stockage et de puissance de calcul tout en assurant une meilleure proximité avec les scientifiques 2

3 A lot of bioinforma-cs tools tools BLAST FastA OMSSA ClustalW2 SSearch PeptideShaker ARIA BWA X!tandem HMMer TopHat samtools Galaxy Clustal Muscle fastqc Omega R ABYSS ARIA 2.3 Bioconductor 2.11 biomaj BLAST Blat 35 Bowtie Bowtie beta7 BWA BWA CAP3 CD-HIT Clustal Omega CLUSTALW 2.1 Cufflinks Cutadapt E-SURGE Exonerate express FastA 3.6 FastQC Galaxy portal GATK HMMer 3.0 ImageJ 1.48 khmer 1.1 M-SURGE MEME 4.7 MMSEQ a Mobyle MODAL MultAlin MUSCLE neo4j Oases OMSSA PeptideShaker phyml 3.1 PREDATOR proline python 2.7 R 2.13 R R R-studio Ray 1.3 RSAT samtools Samtools 1.1 SearchGUI SeqClean Shiny Stacks STAR 2.4.0f1 SuMo v1 TGICL TopHat trim_galore Trinity U-CARE VCFtools Velvet X!tandem XPLOR-NIH

4 Many interfaces 4

5 The French Ins-tute of Bioinforma-cs and its e- infrastructure 5

6 IFB - Ins-tut Français de Bioinforma-que IFB, the French distributed infrastructure for life-science information Mission : to make available core bioinformatics resources to the national/international life science research community. To provide support for national biology programs To provide an IT infrastructure devoted to management and analysis of biological data To act as a middleman between the life science community and the bioinformatics/ computer science research community CNRS UMS3601. Avenue de la Terrasse, Bât Gif-sur-Yvette ELIXIR French Node The European distributed infrastructure for lifescience information To optimize the interactions and coordination between the national level and ELIXIR and other ESFRI infrastructures in biomedical and environmental field, To promote consistency and complementarities between the components offered by the ELIXIR French node and those of other European nodes 6

7 IFB e- Infrastructure Mission : to provide core bioinformatics resources to the life science research community. To set up a French IT infrastructure (cloud) devoted to management and analysis of biological data To provide hardware, data collections and bioinformatics tools To collaborate with international infrastructure (ELIXIR) Current resources A national hub : IFB-core IT resources hosted at CNRS IDRIS SC center A network of regional centers 32 bioinformatics platforms - 15,000 cores - 5 PB Two running clouds : IFB-core and GenOuest Create a federation of clouds for life sciences C C 7

8 Virtualisa-on Virtual machines 1 N } } App App Application OS Matériel P R S Re OS Matériel P R S Re Système d exploitation Hyperviseur Matériel Proc. RAM Stock. Rés. Matériel Proc. RAM Stock. Rés. Physical server 8

9 IFB- core s cloud IFB-core # Compute Cores # TB Storage # TB RAM Max VM size Technology Location Pilot c 256GB StratusLab CNRS-IDRIS, Paris 2016-S1 3, ?144c 3TB? StratusLab ,000 2,000 -?? StratusLab CNRS-IDRIS, Paris CNRS-IDRIS, Paris NGS, imaging, statistics, PaaS IaaS launch jobs Scientists RENATER 10giga SaaS Frontend Master Virtualization Layer Shared FS Workers Web portal Pdisk storage iscsi 10giga eth IDRIS CNRS SC-center 10giga eth Cloud Hypervisors - std nodes: 32c 128GB - bigmem nodes: 40c 256GB 9

10 Provide scien-sts with bioinforma-cs resources - data and tools - as cloud appliances 10

11 Create bioinforma-cs appliances VM 1 VM n Application Application OS OS HW HW Hypervisor HW tools BLAST FastA OMSSA ClustalW2 SSearch PeptideShaker ARIA BWA X!tandem HMMer TopHat samtools Galaxy Clustal Muscle fastqc Omega Create new cloud services Virtual Machines R + Linux system Bioinformatics Marketplace Appliance? predefined virtual machine including tools, pipeline,recipes Ready to run Appliance annotation Title Description + (w. controlled voc.) Topics Tools Contact Developer(s) and maintainer(s)! Structures Sequences Proteomics Galaxy... 11

12 Remote desktop IFB s bioinforma-cs appliances Proteomics Imaging Web Galaxy MODAL Eco Pop Galaxy Galaxy RADseq Galaxy AVIESAN 2015 BioDataCloud IGV Scientific apps RSAT z PhyML MacSyFinder SynBioWatch R CLI statistics biocompute Node Aria biohadoop Utilities biodata BioMaj BlobSeer biodata NFS Cassandra Docker CentOS Ubuntu Base OS Neo4j Data mgmt 12

13 AppS Cloud Galaxy Portal Galaxy portal is widely used in the community analyse NGS data (mainly but not only) connected to community knowledge: data and indexes, tools, workflows Cloud advantages : User is administrator of his/her own Galaxy instance: install data and tools Versions 1,2..n & Updates Preserve workflows and results in cloud storage Help the integration of monthly updates and new tools Cloud permit different appliances to be built from the same base: a basic one with common tools for NGS specific ones for a domain or a set of tools e.g. Galaxy-MODAL, Galaxy-RADseq, EBA-ChIP-Seq for training: create a special appliance with dedicated datasets, tools or workflows e.g. AVIESAN school 2013 manual Galaxy NGS Run Linux system Galaxy + Tools Galaxy RADseq Galaxy automatic Galaxy MODAL Aviesan Cloud Marketplace Install Publish VMs 13

14 App R Sta-s-cal Compu-ng R software environment for statistical computing and graphics include common bioinformatics module RStudio IDE Biobase, BiocGenerics, BiocInstaller, GenomeInfoDb integrated development environment (IDE) for R features: console, syntax-highlighting editor Shiny web framework powerful web framework for building web applications using R. without requiring HTML, CSS, or JavaScript knowledge. Contact: Stéphane Delmotte (PRABI-LBBE) 14

15 IFB s cloud for Bioinforma-cs Public Data sources Data BioMAJ EMBL PDB Genomes UNIPROT PROSITE Reference Datasets common share VMs VMs VMs VMs VMs VMs VMs VMs VMs VMs Cloud Credentials Data Personalized interfaces j. doe e. martin you chb virtual disks cg User data Author. VMs VMs VMs Cloud for Bioinformatics 15

16 A cloud driven through a web dashboard 16

17 Browse the marketplace and run an App! Proteomics Sequences Galaxy Structures?... IFB s bioinformatics marketplace! 17

18 Pra-que 18

19 Connexion au Cloud IFB Connectez-vous au cloud IFB dans Sign in 19

20 Connexion au Cloud IFB (2) Lors de la première connexion rubrique Settings complétez vos paramètres! clé SSH: fichier ~/.ssh/dsa.pub attention aux retours à la ligne lors du copier-coller la créer avec sshkeygen (ou PuTTYgen) 20

21 Le tableau de bord du Cloud IFB Gérer ses VMs Créer / arrêter / renommer Gérer ses disques virtuels Créer / supprimer Visualisez les paramètres nom/type/taille état/charge CPU disque virtuel attaché 21

22 Créer une machine virtuelle Proteomics Sequences Galaxy Structures?... IFB s Marketplace! 22

23 Choisir la bonne appliance 23

24 Caractéris-ques d une VM Définir sa VM un nom nombre de CPUs taille mémoire attacher un disque virtuel Cluster de VMs remplir le nombre de VMs choix du nom unique Appelée aussi Instance 24

25 Les disques durs virtuels Pour stocker ses données taille et nombre variable (quota) retrouver ses données d une VM à la suivante.! pas de sauvegarde! Sur un vdisk l attacher à une (seule) VM à la création de la VM Partager un vdisk mode cluster VM NFS 25

26 Pra-que Depuis le tableau de bord du cloud Créez votre disque virtuel bouton New vdisk monddgalaxy, 10Go 26

27 Pra-que Depuis le tableau de bord du cloud bouton New instance identifier quelle appliance fournit l outil deeptools? quelle version de l outil? Créer une instance EBA15 Galaxy ChIP-seq nom: galaxy_roscoff_2015 taille: c3.medium (2 CPU, 8 GB RAM) attacher votre disque virtuel monddgalaxy 27

28 Monitor your usage 28

29 Ques-ons? Acknowledgments IFB members IFB hub: Patricia, Jean-François, Mohamed, Jonathan, Maxime, Dominique Alumni : Marie, Quentin we are hiring! Working group IFB-GRISBI (co-chair with Olivier Collin) Appliances developers Samuel Blanck (Inria Lille), Jacques van Helden (TAGC), Stéphane Delmotte (PRABI-Doua), Bruno Spataro (PRABI-Doua), Marie-Laure Franchinard (MIGALE), Anis Djari (BioinfoGenoToul), Bertrand Néron (Institut Pasteur), Adrien Josso (MicroScope), Thomas Lacroix (MIGALE), Christian Baudet (CLB), Germain Paimparay & Baptiste Brault (CFB) CNRS IDRIS: R. Medeiros, C. Gauthey and staff StratusLab members IFB is funded by French programs PIA INBS 2012, BioDataCloud EU H2020 projects, CYCLONE (644925) and EGI-Engage (654142) 29