Grille de calcul pour le biomédical Journée BioSTIC, 10 juillet 2008

Transcription

1 Grille de calcul pour le biomédical Journée BioSTIC, 10 juillet 2008 Johan Montagnat Diane Lingrand CNRS / UNSA, I3S laboratory, RAINBOW team EGEE-II INFSO-RI

2 EGEE Grid Infrastructure Flagship European grid infrastructure project Now in 3rd phase with more than 100 partners Size of the infrastructure today: > 250 sites in 48 countries > CPU cores ~ 5 PB disk + tape MSS > jobs/day > 9000 registered users EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

3 User requests Middleware components DataSets info resources info Information Service Data Management Author. &Authen. queries queries Workload Management Site X Resources allocation Publication Sites Resources Indexing Computing Resources Storage Resources Dynamic evolution Logging, real time monitoring EGEE-II INFSO-RI DICOM Image Management, J. Montagnat, HealthGrid, June 2,

4 glite middleware Complex middleware stack On top of Globus Toolkit 2, VDT, CONDOR, batch schedulers... Fundational services User authentication and authorization Information Index: registration of Computing Elements (CEs), Storage Elements (SEs), etc. Workload Management System: workload distribution over sites Data Management System: virtual file hierarchy, standard (SRM) interface to storage resources Batch oriented EGEE is a federation of computing centers and File Servers oriented EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

5 Why grids for medical imaging? Enabling Grids for E-sciencE Sharing computing resources and algorithms Research (populations studies, models design, validation, statistics) Complex analysis (compute intensive image processing, time constraints...) Data Procedures Algorithms S e q 1 > d c s c d s s d c s d c d s c bscdsbcbjbfvbfvbvfbvbvbhvb hsvbhdvbhfdbvfd S e q 2 > b v d f v f d v h b d f v b bhvdsvbhvbhdvrefghefgdscg dfgcsdycgdkcsqkc S e q n > b v d f v f d v h b d f v b bhvdsvbhvbhdvrefghefgdscg dfgcsdycgdkcsqkchdsqhfduh dhdhqedezhhezldhezhfehfle zfzejfv Computing power EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

6 Building on Production Grids Applications Applications Applications Applications Applications level Application-level services (high-level middleware) Application-level services (high-level middleware) Production grid infrastructure level Resources (low-level) Middleware Resources Resources Resources Resources Communication layer (GEANT, Internet...) EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

7 The Biomed Virtual Organization Enabling Grids for E-sciencE Fostering scientific communities EGEE is an international, multi-disciplinary research infrastructure Scientific communities are expanding beyond administrative boundaries Virtual Organizations Authentication and Authorization management Share resources (computers, data, algorithms...) inside a VO Application areas identification and support unit The Biomed VO now divided in 3 sectors Medical imaging Bioinformatics Drug discovery EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

8 Biomed Virtual Organization Size of the infrastructure today: > 250 sites in 48 countries > CPU cores ~ 5 PB disk + tape MSS > jobs/day > 9000 registered users Out of which, Biomed VO: > 100 sites in 30 countries (170 CEs, 130 SEs) ~ CPU > 150 registered users EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

9 Irregular use through time Year 2007 statistics collected per month EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

10 Total production in 2007 ~800 CPU years (1000 SpecInt 2000 Normalised CPU time) EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

11 Total production in 2007 ~2,600,000 jobs EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

12 Computing resources provision Enabling Grids for E-sciencE Computing hours per EGEE federation in 2007 EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

13 Biomed VO share Enabling Grids for E-sciencE Life sciences compared to other sciences CPU time Number of jobs Biomed VO EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

14 Building on Production Grids Applications Applications Applications Applications Applications level Application-level services (high-level middleware) Application-level services (high-level middleware) Production grid infrastructure level Resources (low-level) Middleware Resources Resources Resources Resources Communication layer (GEANT, Internet...) EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

15 WISDOM In silico Drug Discovery Enabling Grids for E-sciencE WISDOM: Goal: find new drugs for neglected and emerging diseases Neglected diseases lack R&D Emerging diseases require very rapid response time Need for an optimized environment To achieve production in a limited time To optimize performances Method: grid-enabled virtual docking Cheaper than in vitro tests Faster than in vitro tests EGEE-II INFSO-RI Application to Courtesy life sciences, of V. J. Breton Montagnat, June 18, 2008

16 Millions of chemical compounds available in laboratories High Throughput Virtual Docking Enabling Grids for E-sciencE Chemical compounds : ZINC Molecular docking : FlexX, Autodock Targets structures : PDB Grid infrastructure : EGEE Chemical compounds : Chembridge 500,000 Drug like 500,000 High Throughput Screening 1-10$/compound, nearly impossible ( Autodock Molecular docking (FlexX, ~80 CPU years, 1 TB data Computational data challenge ~6 weeks on ~1000/1600 computers Targets : ( 1lf3 Plasmepsin II (1lee, 1lf2, ( 1ls5 ) Plasmepsin IV Hits screening using assays performed on living cells Leads Clinical testing Drug EGEE-II INFSO-RI Application to life sciences, J. Montagnat, June 18,

17 The new WISDOM architecture Major new features: - improved data management - secure storage - migration to web service Credit: J. Salzeman, V. Bloch INFSO-RI Healthgrid March 8th, 2007 V. Breton 17

18 WISDOM-II: A huge international effort Significant contributions from several International grid infrastructures 1%2% 2% 3% 3% 3% 39% 3% 5% 6% 7% EGEE Germany Switzerland EGEE Asia Pacific EGEE Russia Auvergrid EuChinaGrid EELA EGEE South Western Europe EGEE Central Europe EGEE Northern Europe EGEE Italy EGEE South Eastern Europe EGEE France EGEE UKI 12% 15% Over 420 CPU years in 10 weeks A record throughput of docked compounds per hour INFSO-RI Healthgrid March 8th, 2007 V. Breton 18

19 The grid added value LPC Clermont-Ferrand: Biomedical grid Enabling Grids for E-sciencE The grid provides the centuries of CPU cycles required on demand The grid provides the reliable and secure data management services to store and replicate the biochemical inputs and outputs The grid offers a collaborative environment for the sharing of data in the research community on avian flu and malaria CEA, Acamba project: Biological targets, Chemogenomics SCAI Fraunhofer: Knowledge extraction, Chemoinformatics Univ. Modena: Biological targets, Molecular Dynamics Chonnam nat. univ.: In vitro testing New HealthGrid: Biomedical grid, Dissemination Univ. Los Andes: Biological targets, Malaria biology Univ. Pretoria: Bioinformatics, Malaria biology ITB CNR: Bioinformatics, Molecular modelling Academica Sinica: Grid user interface Biological targets In vitro testing EGEE-II INFSO-RI Application to Courtesy life sciences, of V. J. Breton Montagnat, June 18,

20 Statistics of deployment First Data Challenge: July 1st - August 15th 2005 Target: malaria 80 CPU years 1 TB of data produced 1700 CPUs used in parallel 1st large scale docking deployment world-wide on a e-infrastructure Second Data Challenge: April 15th - June 30th 2006 Target: avian flu 100 CPU years 800 GB of data produced 1700 CPUs used in parallel Collaboration initiated on March 1st: deployment preparation achieved in 45 days Third Data Challenge: October 1st - 15th December 2006 Target: malaria 400 CPU years 1,6 TB of data produced Up to 5000 CPUs used in parallel Very high docking throughput: > compounds per hour EGEE-II INFSO-RI Application to Courtesy life sciences, of V. J. Breton Montagnat, June 18,

21 Scientific objectives Molecular Bioinformatics: protein sequence analysis Analyse data from high-throughput Biology: complete genome projects, EST, complete proteomes, structural biology,. Integration of biological data and tools Method Provide Biologists with an usual Web interface: Web portal online since tools & 12 updated databases + 10,000,000 jobs & 5,000 jobs/day Ease the access to updated databases and algorithms. Protein databases are stored on grid storage as flat files. Legacy bioinformatics applications Wrapping usual binary in grid environment transparent remote access with local filesystem Display results in graphical Web interface. Status: Production Contact: Christophe.Blanchet@ibcp.fr GPSA: Bioinformatics Grid Portal EGEE-II INFSO-RI Application to Courtesy life sciences, of C. Blanchet J. Montagnat, June 18,

22 GPSA Results Enabling Grids for E-sciencE Grid-enabled bioinformatics resources 9 algorithms 3 protein databases Bioinformatics descriptors XML framework, Web portal, Web Services Encryption system: EncFile Security: AES, Key sharing, M-of-N Transparent access to grid files: Parrot EGEE-II INFSO-RI Application to Courtesy life sciences, of C. Blanchet J. Montagnat, June 18,

23 Pharmacokinetics UPV (University Polytechnic of Valencia) GATE LPC (CNRS Clermont Ferrand) SiMRI3D, ThIS, CAVIAR CREATIS (CNRS - Lyon) gptm3d LRI-LAL-LIMSI (CNRS - Orsay) Bronze Standard I3S (CNRS Sophia), INRIA SPM Alzheimer's U. of Genova / DEST EGEE Medical imaging sector LRMN EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

24 Pharmacokinetics: contrast agent diffusion Scientific objectives Study of contrast agent diffusion Computations Image sequences co-registration (independent computations) Parallel contrast agent diffusion modeling Features Application portal for use in clinical environment Enable on-line analysis of exams acquired daily Examination time ~15 minutes Exam analysis time ~2 hours 30 minutes EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

25 Scientific objectives Better understand MR physics. Study MR sequences in-silico. Study MR artefacts. Validate MR Image processing algorithms on synthetic yet realistic images. Method Simulate Bloch's electromagnetism equations. Parallel (MPI) implementation to speedup computations. SiMRI3D: MRI simulator LRMN D image simulation: ~100 CPU years. EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

26 Open GATE collaboration Scientific objectives Geant4 Application for Tomographic Emission Medical physics applications: PET camera simulation, radiotherapy, occular brachytherapy treatment Method GEANT4 base software to model physics of nuclear medicine. Use Monte Carlo simulation to improve accuracy of computations (as compared to the deterministic classical ( approach EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

27 GATE application to radiotherapy Enabling Grids for E-sciencE Accurate dose computation for radiotherapy planning Scanner slices: DICOM or DICOMRT format EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

28 Scientific objectives User-assisted reconstruction of 3D anatomical structures Quantification (e.g. volume measurements) Use models for augmented reality Computations Parallleization of segmentation processes Large number of short jobs Features ( real-time Highly interactive (almost Used in clinical environment Windows interface gptm3d: radiology analysis EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

29 MR Images collection (T1, T2, PD) Preprocessing Nonuniformity correction Intensity normalisation Use case: multiple sclerosis Software technologies for integration of process, data and knowledge in medical imaging Registration and resampling Stereotaxic registration Resampling Segmentation Brain segmentation Classification Segmentation 4 classes: - White matter - Gray matter - CSF - lesions Multimodal MRI Classification Quantification Statistical tests NeuroLOG ANR-06-TLOG-024 Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, Analysis

30 Large databases 256 3D images (test database) Experiments requirements Software technologies for integration of process, data and knowledge in medical imaging 120 3D images (mono-site clinical database) D images (multi-site clinical trial, TBs of data) Various computing tools 10 to 15 processing stages in the pipeline Computing power > 2 months sequential execution time Pipeline description and execution Workflow description Workflow manager (execution monitoring, restart on error...) NeuroLOG ANR-06-TLOG-024 Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

31 Application to rigid registration algorithms evaluation Enabling Grids for E-sciencE T O 1 O 2 Unregistered Registered EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

32 ~100 image pairs ~800 EGEE jobs Enabling Grids for E-sciencE Bronze standard workflow A B CrestLines Params Params Params Service Params Params Params PFMatchICP GetFromEGEE Yasmina Baladin PFRegister Params FormatConv GetFromEGEE GetFromEGEE GetFromEGEE WriteResults FormatConv FormatConv FormatConv WriteResults WriteResults WriteResults MethodToTest MultiTransfoTest EGEE-II INFSO-RI Accuracy Translation Accuracy Rotation Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

33 Embarassingly parallel problem Fast turn over of short jobs Data confidentiality Interactivity Fine grain parallelism (MPI) Workflow-based Portals / user interface Application specific requirements EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

34 Building on Production Grids Applications Applications Applications Applications Applications level Application-level services (high-level middleware) Application-level services (high-level middleware) Production grid infrastructure level Resources (low-level) Middleware Resources Resources Resources Resources Communication layer (GEANT, Internet...) EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2, 2008

35 WAN medical image exchange Cross-enterprise exchange of radiology reports and images Wide-area Health Information Network Radiology Enterprise B PACS B Radiology-to-radiology Physician Office Radiology-to- Physicians PACS A Radiology Enterprise A EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

36 Grid technologies Wide-area Health Information Network WAN medical image exchange Radiology Enterprise B PACS B Cross-enterprise authentication Access control Secured communications Efficient and reliable transfer PACS A Physician Office Radiology Enterprise A EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

37 Medical Data Manager Objectives Expose a standard grid interface (SRM) for medical image servers (DICOM) Use native DICOM storage format Fulfill medical applications security requirements Do not interfere with clinical practice DICOM Interface SRM Worker Nodes DICOM clients DICOM server User Interfaces EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

38 Medical Data Registration 1. Image is acquired 2. Image is stored in DICOM server 4. image metadata are registered AMGA Metadata 3. lcg client GFAL API 3a. Image is registered (a GUID is associated) 3b. Image key is produced and registered Hydra Key store LCG File Catalog EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

39 Medical Data Retrieval 1. get GUID from metadata AMGA Metadata User Interface Worker Node 7. get file key and decrypt file locally 2. lcg client 6. on-the-fly encryption and anonimyzation return encrypted file Key ACL control Hydra Key store GFAL API 5. get file key SRM-DICOM interface LCG File Catalog 3. get SURL from GUID 4. request file Metadata ACL control File ACL control Anonimization & encryption DICOM server EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

40 Grid Workflow Efficient Enactment for Data Intensive Applications Application workflows representation and efficient enactment on grids GWENDIA ANR-06-MDCA-009

41 Grid Workflow Efficient Enactment for Data Intensive Applications MOTEUR, I3S laboratory, CNRS High level interface Hides grid complexity to the user Service-based approach Legacy code service wrapper Scufl language (mygrid / Taverna) Pure data flow approach Grid submission interfaces EGEE (LCG2, glite) Grid5000 (OAR, DIET) Transparent parallelism exploitation Code and data parallelism Workflow management GWENDIA ANR-06-MDCA-009 Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

42 Efficient parallel execution Grid Workflow Efficient Enactment for Data Intensive Applications A workflow naturally provides application parallelization MOTEUR transparently exploits 3 kinds of parallelism Workflow parallelism Data parallelism Service parallelism D 0, D 1 D 0, D 1 D 0, D 1 S 1 S 1 S 1 D 1 D 0 D 1 S 2 S 3 S 2 S 3 S 2 DS 30 Jobs grouping strategy in sequential branches in order to reduce grid latency MOTEUR GWENDIA ANR-06-MDCA-009 Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,

43 Conclusions Using the European grid infrastructure for production Regular usage in medical imaging community Sustained production since 2004 Closer to clinical end-users RSNA'07 demonstration (EGEE-MEDICUS) Molecular docking for drug discovery Higher level services More data protection (Role-based access control) Considering multiple actors involved in clinical procedures (patient, doctor, surgeon, radiologist, nurse, anesthetist..) Data semantics Metadata associated to data, Ontologies, Content-based data identification, indexing, mining... User interfaces Integrated in end-users environments; often specific to applications EGEE-II INFSO-RI Medical image analysis on EGEE, J. Montagnat, BioGrid, June 2,