Challenges of Managing Scientific Workflows in High-Throughput and High- Performance Computing Environments

Transcription

1 Challenges of Managing Scientific Workflows in High-Throughput and High- Performance Computing Environments Ewa Deelman USC Informa1on Sciences Ins1tute h8p:// Funding from DOE, NSF, and NIH

2 Community Archives: Galactic Plane Atlas 2015! 18 million input images (~2.5 TB)! 900 output images (2.5 GB each, 2.4 TB total)! Measuring the global star formation rate in the galaxy! Studying the energetics of the interaction of molecular clouds with the interstellar medium! Determining whether coagulation or fragmentation governs the formation of massive stars! Assessing the supernova rate in the Galaxy Bruce Berriman John Good

3 Southern California Earthquake Center CME 2010: CyberShake 1.0, the world s first physics-based probabilistic seismic hazard map. Tom Jordan CyberShake Scott Callaghan Phil Maechling

4 2011 Duncan Brown Peter Couvares

5 Improving Soybean Productivity: SoyKB, Results made available to the community SNP and indels calling,. quality assessment, genomic annotation, etc. Trupti Joshi

6 Outline! Pegasus From Virtual Data to Workflows! Challenges in Managing Workflows in Distributed Environments! Challenges in Managing Workflows in HPC systems! Conclusions

7 GriPhyN Project , the beginning of Pegasus Carl Kesselman Ian Foster Miron Livny Paul Avery

8 LIGO Prototype 2001 Gaurang Mehta Recipe of how to generate virtual data " workflow Kent Blackburn Albert Lazzarini Roy Williams

9 Demonstration at SC 2001 From Virtual Data Concepts to Design decisions Abstract Workflows Resource discovery Planning Separation of planning and execution Leveraged remote job submission

10 First Mention of Pegasus in Print 2002 Jim Blythe Yolanda Gil Exploration of using AI planning technologies for workflow mapping, semantic template-based composition, metadata propagation

11 2002: Pegasus: the concept of data reuse Karan Vahi Introduced the notion of data reduction And later workflow-level checkpointing

12 Leveraging existing technologies! DAGMan workflow execution engine, since day 1! Keeps track of job dependencies! Through Condor schedd/condor-g submits jobs to remote resources Manages individual job execution Provides detailed execution logs! Scalability: can handle millions of jobs Miron Livny Pegasus Grants: 2007 and 2012 (NSF)! Reliability: Job retries Rescue DAG! Provides user support Kent Wenger

13 2002: National Virtual Observatory Alex Szalay JHU Input Developed solutions for uniform access to astronomy archives Developed and promoted tools to support astronomy data analysis Montage a major NVO tool, important workflow benchmark Jim Gray Bruce Berriman Science-grade Mosaic of the Sky Reprojection Background Rectification Co-addition Output John Good Image1 Project Background Diff Fitplane Image2 Project BgModel Background Add Diff Fitplane Image3 Project Background Montage Workflow

14 Workflows can be simple! J1 J2 J3 J4 J5 J6 J7 J8 J9 Jn

15 Science-grade Mosaic of the Sky Size of mosaic in degrees square Number of input data files Number of tasks Number of intermediate files Amazon M1 large with 2 cores Total data footprint Cummulative wall time GB 11 mins GB 43 mins GB 1 hour, 56 mins GB 3 hours, 42 mins GB 6 hours, 45 mins

16 Workflows have different computational needs SoCal Map needs 239 of those MPI codes ~ 12,000 CPU hours, Post Processing 2,000 CPU hours Data footprint ~ 800GB

17 Desired Workflow Management Properties! Submit locally, run globally! Automate to the max! Make use of the right resources! Provide reliability and performance time to solution! Help inspect the results! Provide performance information

18 Outline! Scientific Workflows! Pegasus From Virtual Data to Workflows! Pegasus Today! Challenges in Managing Workflows in Distributed Environments! Challenges in Managing Workflows in HPC systems! Conclusions

19 Related Work Workflow systems [Goble et al 2007] [Ludaescher et al 2007] [Bavoil et al 2005] [Marru et al 2007] [Mesirov et al 2009] [Rex et al, 2003] [Taylor et al 2005] [Yu et al 2010] [Zhao et al 2007] Unique contribution: Planning and Optimization of workflow execution

20 Our Approach # Analysis Representation # Support a declarative representation for the workflow # Represent the workflow structure as a Directed Acyclic Graph (DAG) # Tasks operate on files # Use hierarchical representations to achieve scalability # System (Plan for the resources, Execute the Plan, Manage tasks) # Layered architecture, each layer is responsible for a particular function # Mask errors at different levels of the system # Modular, composed of well-defined components, where different components can be swapped in # Use and adapt existing graph and other relevant algorithms

21 Our Approach: Submit locally, Compute globally Data Storage Work definition data Campus Cluster NSF XSEDE DOE Leadership-class Systems Open Science Grid Workflow Management System work Academic Clouds: FutureGrid, Cameleon, etc Amazon/Google Cloud Local Resource

22 Pegasus Workflow Management System! A workflow compiler! Input: abstract workflow description, resource-independent! Auxiliary Info (catalogs): available resources, data, codes! Output: executable workflow with concrete resources! Automatically locates physical locations for both workflow tasks and data! Transforms the workflow for performance and reliability! A workflow engine (DAGMan)! Executes the workflow on local or distributed resources (HPC, clouds)! Task executables are wrapped with pegasus-kickstart and managed by Condor schedd! Provenance and execution traces are collected and stored! Traces and DB can be mined for performance and overhead information

23 Outline! Pegasus From Virtual Data to Workflows! Pegasus Today! Challenges in Managing Workflows in Distributed Environments! Challenges in Managing Workflows in HPC systems! Conclusions

24 Challenges in High-Throughput Infrastructures! Failures in the execution environment or application! Data storage limitations on execution sites! Performance Small workflow tasks! Heterogeneous execution architectures Different file systems (shared/non-shared) Different system architectures (Cray XT, Blue Gene, ) Mismatch between tasks and architecture

25 Generating executable workflows (DAX) APIs for workflow specification (DAX--- DAG in XML) Java, Perl, Python Information Catalogs 25

26 Data reuse for Performance and Reliability Sometimes intermediate data is already available Want to restart the workflow from where it left off f.ip f.ip f.ip A A A f.a B f.a C f.a B f.a C f.a C Workflow Reduction f.b D f.c E f.b D f.c E f.c E Data Reuse f.d F f.e f.d F f.e f.d F f.e Workflow-level checkpointing f.out f.out f.out Abstract Workflow File f.d exists somewhere. Reuse it. Mark Jobs D and B to delete Delete Job D and Job B

27 Storage limitations Small amount of space Gurmeet Singh Automatically add tasks to clean up data no longer needed LIGO was running on Open Science Grid resources, processing TBs of data within a single workflow

28 Montage Astronomy Workflow Rizos Sakellariou 1.25GB versus 700 MB Arun Ramakrishnan

29 Full workflow: 185,000 nodes 466,000 edges 10 TB of input data 1 TB of output data. 166 nodes LIGO Workflows Need additional restructuring 26% improvement 56% improvement

30 Storage limitations Variety of file system deployments: shared vs non-shared User workflow Mats Rynge Allows us to run easily on Open Science Grid, Clouds

31 Workflow Restructuring to improve application performance! Cluster small running jobs together to achieve better performance! Issues Each job has scheduling overheads Execution sites have limits on the number of job submissions Clustered tasks can reuse common input data less data transfers A A B C B C B C B C B C B C B C B C Level-based clustering Label-based clustering Time-based clustering D D Gurmeet Singh

32 Explored Different Clustering methods via Simulation and experimentation with Pegasus! WorkflowSim, available on github! Mimics a Workflow Management System Workflow Mapper, Workflow Engine, Clustering Engine, Workflow Scheduler To support research in robustness! System Overhead Red.! Monetary Cost! Energy usage Weiwei Chen

33 Southern California Earthquake Center % Description CyberShake PSHA Workflow $ Builders ask seismologists: What will the peak ground motion be at my new building in the next 50 years? $ Seismologists answer this question using Probabilistic Seismic Hazard Analysis (PSHA) 239 Workflows! Each site in the input map corresponds to one workflow! Each workflow has: $ 820,000 tasks 2009

34 Evolution of SCEC CyberShake Probabilistic Seismic Hazard Analysis: What will the peak ground motion be at my house in the next 50 years? Each map = 286 sites, 30M SUs Titan (pilots) Blue Waters

35 CyberShake: Computing Needs Over Time 16 Million Files

36 A A Solutions B B B B B B B B Cluster tasks C C C C C C C C tasks Pilot Job D Use pilot jobs to dynamically provision a number of resources at a time D time Develop an MPI-based workflow management engine to manage sub-workflows

37 Custom Workflow Engines Needed for Different Execution Sites! For single tasks! For clustered tasks Gideon Juve! For non-shared file system environments! Pegasus MPI-Cluster for HPC systems A master/worker task scheduler for running fine-grained workflows on batch systems Runs as an MPI job Uses MPI to implement master/worker protocol Works on most HPC systems Requires: MPI, a shared file system, and fork() Allows sub-graphs of a Pegasus workflow to be submitted as monolithic MPI jobs to remote resources

38 CyberShake: Addressing Computing Challenges Hierarchical Workflows, Task Clustering/ Glideins + Corral frontend Hierarchical Workflows, PMC, merged SeisPSA jobs, in-memory rupture variation generation Use of GPU and CPU executables PMC Core days Makespan NCSA Mercury TACC Ranger USC/HPCC Ranger NICS Kraken Blue Waters Blue Waters Stampede Blue Waters

39 Pegasus Releases: Interleaving Research and Software Development Nightly builds Demonstra*on at SC'02 Support for GT4 Bug fixes New par**oning and clustering First standalone Pegasus release Ini*al AWS Cloud support Hierachical workflows User no*fica*ons, bener debugging tools Pegasus MPI- cluster Online monitoring dashboard Builds done using Docker Development started Research: Data Cleanup, 2003 Data footprint: 2007, 2013 Cloud: 2008 Data Cleanup Support for RC and MDS Stable release Workflow Par**oning Task clustering, Stable release Moved to NMI B&T Pegasus Lite Ensemble Manager, Google cloud Major performance improvements Moved code to GitHub New data transfer tools, pegasus- sta*s*cs, pegasus- plots Python API for crea*ng workflows

40 Outline! Pegasus From Virtual Data to Workflows! Pegasus Today! Challenges in Managing Workflows in Distributed Environments! Challenges in Managing Workflows in HPC systems! Conclusions

41 Trends in HPC scaling that will affect applications and runtime software! Increased power awareness Future HPC systems need to reduce power consumption (O(100))! Deep memory hierarchies DRAM, NVRAM, SSD, HDD, long-term storage Increased gap between memory access time and FLOPS! Heterogeneous architecture Billions of elements CPUs and GPUs! Novel I/O solutions as part of overall architecture Burst buffers (storage with computing attached)! More faults Would need more checkpointing, but that s expensive

42 Increased power awareness! Data movement is a significant part of the overall energy consumption! Data movement within the HPC system needs to be carefully coordinated Implies that data delivery to/from the system needs to be poweraware as well Need to develop new algorithms that take into data locality on the system! Some computations need to be done in-situ without writing to storage In-situ workflows which coordinate processing and visualization SCEC: in-situ post-processing Need to decide which data to keep, impacts provenance as well

43 What we learned in Distributed Area WMS We can apply to HPC application management! Reliability: WMS deal with: task failures, problems accessing data, resource failures, and others. Investigate how data replication techniques can be used to improve fault tolerance, while minimizing the impact on energy consumption Explore tradeoffs between data re-computation and data retrieval from DRAM/NVM/disk (time to solution and energy consumption)! Provenance Capture and Reproducibility: WMS capture provenance information about the creation, planning, and execution Up to now, the approach has been to save everything problem Provenance capture may need to adapt to the behavior of the application (coarse and fine levels of details, compression) May want to automatically re-run parts of the computation and reproduce the results and a more detailed provenance trail on demand! What we need: Workflow/Applications Performance/Behavior Modeling across scales

44 Issues in Workflow Performance Modeling and Prediction! Workflows involve Diverse applications A number of heterogeneous, distributed resources (compute, storage, networks) A layered software stack: Workflow engine, local scheduler, remote scheduler, runtime system Wide-area data movement services, on-site file system, memory! Varied and sometimes sparse monitoring systems Usually good network data Usually good low-level performance data on compute nodes Poor scheduler, data storage data! Lack of end-to-end performance/resource usage models! Lack of benchmarks that exercise the end-to-end system! Lack of traces that capture the workflow characteristics

45 To advance the Science of Workflows we need! Workflow/Applications Performance/Behavior Modeling: Understand the resource needs and behavior (performance, energy usage) of the workflow applications across scales: workflow ensemble, workflow instance, down to individual tasks and code segments Need community benchmarks, execution traces and profiles a beginning, has 11 workflow applications, most with multiple runs, synthetic workflow generator, simulator for the wide area Beginning of a performance archive Methodologies for predicting the behavior of applications On current infrastructures On future infrastructures That can inform future infrastructures! Fundamentally new software and software compositions that can seamlessly work across the scales, particularly in the data management area (wide area networks, deep memory hierarchies)

46 To Advance Computing for Science we need to Enhance the ease of use of software tools Can you do science as an app? Should we take a new look at Virtual Data? Still much more to do!

47 How to make progress?! Apply your knowledge to real world problems! Abstract real problems to generate new techniques and knowledge! Interleave software development and research! Test ideas in real settings! Exercise patience with yourself, explain the need for patience to funding agencies