REPRODUCIBILITY OF EXECUTION ENVIRONMENTS IN SCIENTIFIC WORKFLOWS USING SEMANTICS AND CONTAINERS

Transcription

1 REPRODUCIBILITY OF EXECUTION ENVIRONMENTS IN SCIENTIFIC WORKFLOWS USING SEMANTICS AND CONTAINERS Rafael Ferreira da Silva, Ewa Deelman USC Information Sciences Institute Container Strategies for Data & Software Preservation that Promote Open Science May 19-20, University of Notre Dame, USA

2 OUTLINE Introduction Semantic Modeling Semantic Annotations Motivation Scientific Workflows Conservation Annotations Virtualization WICUS Semantic Models Computational Reproducibility PRECIP Pegasus Reproducibility Process Infrastructure Specification Algorithm Use Cases Summary Conclusions Future Research Directions 2

3 Scientific Workflows Large scale computations Provenance INTRODUCTION DAG directed-acyclic graphs Reproducibility Data, code, workflow description Underlying infrastructure? job Command-line programs Conservation Physical: real object Logical: object description (semantics) dependency Usually data dependencies 3

4 SEMANTIC MODELING Former Equipment Desktop Cluster Grid Cloud Equivalent Execution Envrionment Virtual Images Containers Reproducibility Timeline ANNOTATE REPRODUCE Semantic Annotations Workflow Software Hardware Computing Resources 4

5 SEMANTICS WICUS Workflow Infrastructure Conservation Using Semantics Ontology Network 5

6 EXPERIMENT MANAGEMENT PRECIP Pegasus Repeatable Experiments for the Cloud in Python Overview Experiment management control API Works with commercial and academic clouds Tag-based system Features No need of pre-installed software on the VM image Create VM, transfer files Run commands remotely (SSH) 6

7 WORKFLOW MANAGEMENT SYSTEM Pegasus WMS Automates complex, multi-stage processing pipelines Enables parallel, distributed computations Automatically executes data transfers Handles failures with to provide reliability Reproducibility Reusable, aids reproducibility Records how data was produced (provenance) Keeps track of data and files job dependency DAG in XML split merge pipeline 7

8 REPRODUCIBILITY PROCESS DAX Annotator Annotations SW Comp Catalog Information Specification Algorithm TC Annotator & Config Annotations DAX XML Pegasus Trans. Catalog SVA Catalog PRECIP Script 8

9 DEPENDENCY MANAGEMENT Infrastructure Specification Algorithm (ISA) Obtain an specification defining what VMs need to be created, what software components must be deployed and their configuration REQ1 SVA1 Workflow REQ2 SVA2 REQ3 SVA3 9

10 USE CASES Workflow WMS Requirements Software Montage Epigenomics SoyKB mfitplane mfit mdifffit mproject... mdiff SolexFilter... FastqSplit Dedup... Haplotype GLibC Maq Map requirement requirement requirement requirement requirement requirement Maq Index Hardware C. Resources CentOS 6 Perl5 LibStd Sol2Sanger CPU RAM DISK CPU RAM DISK CPU RAM DISK AWS CentOS 6 HW FG CentOS 6 HW Vagrant CentOS 6 HW CPU RAM DISK CPU RAM DISK AWS CentOS 6 40GB disk HW FG CentOS 6 40GB disk HW AWS CentOS 6 5GB disk FG CentOS 6 5GB disk CentOS 6 SVA Vagrante CentOS 6 50GB disk AWS CentOS 6 40GB disk FG CentOS 6 40GB disk Sw Bundle Java Wget BWA Gtak Pegasus WMS HTCondor 10

11 USE CASES Montage Epigenomics Astronomy Workflow Montage software distribution 59 binaries (executables) Bioinformatics Workflow 8 binaries (executables) 11

12 SUMMARY Summary Conclusion Future Research Directions Contributions Semantic vocabularies to describe the execution environment of workflows Process for documenting the workflow application and the WMS Applicability to public, private, and local clouds (EC2, Chameleon, Vagrant, and Docker) Future Work Multi-node infrastructure Automation of the semantic annotation process I. Santana-Perez, R. Ferreira da Silva, M. Rynge, E. Deelman, M. S. Pérez-Hernández, and O. Corcho, Reproducibility of Execution Environments in Computational Science Using Semantics and Clouds, Future Generation Computer Systems,

13 REPRODUCIBILITY OF EXECUTION ENVIRONMENTS IN SCIENTIFIC WORKFLOWS USING SEMANTICS AND CONTAINERS Thank You Questions? Rafael Ferreira da Silva, Ph.D. Computer Scientist, USC Information Sciences Institute