DTI Image Processing Pipeline and Cloud Computing Environment

Transcription

1 DTI Image Processing Pipeline and Cloud Computing Environment Kyle Chard Computation Institute University of Chicago and Argonne National Laboratory

2 Introduction DTI image analysis requires the use of many tools QC Registration ROI Marking Fiber Tracking QC, Registration, ROI Marking, Fiber Tracking,.. Constructing analyses is challenging Data & tooldiscovery, selection, orchestration,.. We have made huge strides in terms of data Data formats, repositories, i protocols, metadata, CDEs We now need infrastructure to reduce the barriers that exist between data providers, tool developers, researchers, and clinicians Big Science. Small Labs o We have exceptional infrastructure for the 1%, what about the 99%? 2 DTI Pipelines and Cloud Infrastructure

3 Common Approach to Analysis Modify Install (Re)Run Script Camino 3 DTI Pipelines and Cloud Infrastructure

4 How can we improve? We need a platform where users can easily construct ct and execute eec teanalyses Using best of bread tools and pipelines Abstracting low level infrastructure and platform heterogeneity Supporting automation and parallelism Supporting experimentation => Make existing tools and common analyses mundane building blocks 4 DTI Pipelines and Cloud Infrastructure

5 DTI Metric Reproducibility Pipeline Ultimate Goal: Investigate the feasibility of using DTI in clinical practice Automatic calculation of DTI metrics (FA, MD) from 48 automatically generated ROIs Using existing tools to create a reusable analysis workflow that can be easily repeated Investigate the ability to scale analyses over large datasets Explore the reproducibility over a group of 20 subjects with 4 scans spread over 2 sessions 5 DTI Pipelines and Cloud Infrastructure

6 DTI Processing Pipeline (1) BVEC & BVAL DTI T1 1. ECC DTI (FSL) 2. BET DTI (FSL) 3. BET T1 (FSL) 4. Linear Registration DTI / T1 (FSL FLIRT) 5. DTI Fitting (FSL/Camino) 7. Linear Registration T1/Template (FSLFLIRT) FLIRT) 7. Non linear Registration T1/Template (FSLFNIRT) FNIRT) Template 9. Transform FA/MD to MNI space (FSL Applywarp) p) 8. Calculate ROI Mean FA/MD (AFNI 3dmaskave) Atlas Mask 6 DTI Pipelines and Cloud Infrastructure

7 DTI Processing Pipeline (2) DTI BVEC & BVAL 1. ECC DTI (FSL) 2. BET DTI (FSL) FA Template FA image 3. DTI Fitting (FSL/Camino) Linear Registration (FSL FLIRT) Non Linear Registration (FSL FNIRT) MD image Apply Warp coefficient 7 DTI Pipelines and Cloud Infrastructure FA in MNI space 6. Calculate ROI Mean (3dmaskave) MD in MNI space Atlas Mask

8 Approaches for Implementing Pipelines Scripts XNAT Pipeline Engine Globus Genomics Bash scripts written to Defined by code (XML SaaS for genomics execute tools on a + scripts) Graphical interface for single computer Overhead to include creation and execution Time consuming, error prone, hard to transfer knowledge tools, develop interfaces and create pipelines Supports ondemand provisioning based on pricing gpolicies Little support for parallelization Difficult to change tools/pipelines Tools installed dynamically when Some support for parallelization li required 8 DTI Pipelines and Cloud Infrastructure

9 DTI Pipeline Platform Globus Endpoints Galaxy & Manager Globus Transfer Galaxy Condor Shared File System Dynamic Scheduler Dynamic Worker Pool 9 DTI Pipelines and Cloud Infrastructure

10 DTI Pipelines in the Cloud NFS Gluster GridFTP Condor Schedule 10 DTI Pipelines and Cloud Infrastructure Camino

11 DTI Pipelines in Galaxy 11 DTI Pipelines and Cloud Infrastructure

12 Cloud Computing Leverages economies of scale to facilitate utility ty models Pay only for resources used 1 * 100 hours == 100 * 1 hour On demand and elastic access to unlimited capacity Addresses fluctuating requirements Software as a Service Web access to data through Platform as a Service defined interfaces Platform as a Service No management of hardware or low level tools Infrastructure as a Service 12 DTI Pipelines and Cloud Infrastructure

13 Challenges Moving to the Cloud Resource Selection: Comparing price, capabilities, performance, instance types (EBS, S,Instance store), toolperformance Tool Selection and Management: Finding tools, installing, configuring and using them in different environments Analysis/Resource Management: Developing structured and repeatable analyseswith different tools. Data transfer: Moving large amounts of data in/out of Cloud environmentreliably reliably andefficiently Scale and Parallelism: Scaling analyses by efficiently parallelizing across elastic infrastructure Security: Data and computation security HIPAA? 13 DTI Pipelines and Cloud Infrastructure

14 Amazon EC2 Pricing System Specifications Pricing CPU Units CPU Cores Memory On Demand Spot (Low) Spot (High) m1.large m1.xlarge m3.xlarge m3.2xlarge m2.xlarge m2.2xlarge m2.4xlarge DTI Pipelines and Cloud Infrastructure

15 Spot Pricing Volatility 15 DTI Pipelines and Cloud Infrastructure

16 Instance Performance and Pricing Time (M Minutes ) EBS Instance Store On Demand Spot (Low) Spot (High) Cos st per Subjec ($) 16 DTI Pipelines and Cloud Infrastructure

17 Pricing Multiple Analyses Per Node Co ost per Subject ($) On Demand Spot (Low) Spot (High) 17 DTI Pipelines and Cloud Infrastructure

18 Elastic Startup Cost 1:15:00 Tim me 1:00:00 0:45:00 0:30:00 ROI Calculation Tensor Fitting ECC & Registration Contextualize 0:15:00 0:00:00 New Worker Existing Worker Spot Price Queue 18 DTI Pipelines and Cloud Infrastructure

19 Data Transfer with Globus Online Reliable file transfer, sharing, syncing. Easy fire and forget file transfers Automatic fault recovery High performance Across multiple l security domains In place sharing of files with users and groups No IT required. Software as a Service (SaaS) o No client software installation i o New features automatically available 19 DTI Pipelines and Cloud Infrastructure

20 Transfer Comparison 20 DTI Pipelines and Cloud Infrastructure

21 Summary Structured pipelines simplify creation, execution and sharing of complex analyses and sharing of complex analyses Hosted as a service can further reduce barriers By outsourcing pipeline execution on the Cloud we can reduce overhead and costs Previously we took weeks to process ~100 scans o Using this approach < 5 cents a subject ($5 for 1 hour) What's next? Can we deliver this as a service? o Billing, security, paradigm shift, interactive tools Developing toolsheds for sharing tools and pipelines pp 21 DTI Pipelines and Cloud Infrastructure

22 Acknowledgements Mike Vannier, Xia Jiang, Farid Dahi Globus Online Ian Foster, Steve Tuecke, Rachana Ananthakrishnan Globus Genomics o Ravi Madduri, Paul Dave, Dina Sulakhe, Lukasz Lacinski 22 DTI Pipelines and Cloud Infrastructure