Contrail Summerschool Almere, July 22 26, The UberCloud HPC Experiment Paving the way to HPC as a Service

Transcription

1 Contrail Summerschool Almere, July 22 26, 2013 The UberCloud HPC Experiment Paving the way to HPC as a Service

2 It all started with an observation Business Clouds are becoming widely accepted.. but acceptance of clouds in HPC & Digital Manufacturing is still in an early adopter stage 2

3 The Problem In many industries HPC is needed to stay competitive, but: For SMEs buying and using large scale HPC systems is expensive and complex so they use low end HPC or PC workstations However, 50 % are dissatisfied with their computing capacity * => HPC as a Service (in the Cloud) is an option, but there are roadblocks! * Source:

4 Recent Poll about Roadblocks The UberCloud, June 13, 2013

5 The UberCloud HPC Experiment An open voluntary collaborative community Objective: Making HPC as a Service available, for everybody, on demand How? For SMEs and their engineering applications to explore the end-to-end process of using remote computing resources, as a service, on demand, at your finger tip, and learning how to resolve the roadblocks.

6 The End-User Benefits free, on-demand access to hardware, software, and expertise, with a one stop resource shopping experience No hunting for resources in the complex emerging cloud market Professional match-making of end-users with service providers Perfectly tuned end-to-end, step-by-step process to the HPC Cloud Lowering barriers & risks for frictionless entry into HPC in the Cloud Crowd sourcing: End-Users are building relationships with other community members who actively contribute to improvements

7 How does the Experiment work? End-User joins the experiment ISV joins We select a Team Expert We suggest a Resource Provider Team is ready to go 22 steps on Basecamp virtual team office Finally, writing the Case Study

8 Our recipe Find the best matches and form the team Step by step process Collaborative tools Engaged team participants Experienced mentors Exhibit list of additional services to pick from Community get help when needed

9 Step by Step process Basecamp project management platform for each team

10 Where are we with the experiment Started last August: currently 500+ participating organizations and individuals Participants are from 48 countries 80 teams have been formed in Rounds 1-3 Round 4 starts August 1: already 15 teams Registration at:

11 Participants Some of our 40+ Resource Providers Media Sponsor Some of our Providers want to be anonymous

12 Participants Some of our 40+ Software Providers Some of our ISVs want to be anonymous

13 Participants Some of our 240+ HPC Experts Some of our HPC Experts want to be anonymous and 45 more.

14 UberCloud Exhibit And our media sponsor:

15 Announcements UberCloud HPC Experiment Compendium A collection of 25 selected use-cases from 60 teams in Rounds 1 & 2 Sponsored by Intel s HPC Organization (Thank You!) Professionally edited by Tabor Communications Google ubercloud compendium One-stop UberCloud Exhibit with 25 exhibitors UberCloud University: Lectures about Cloud technologies and applications Round 4 starting in August 2013 Open for registration and project 15 submissions NOW

16 Teams, it s all about teams 20 teams from Round 1: Anchor Bolt Resonance Radiofrequency Supersonic Liquid-Gas Wing-Flow Ship-Hull Cement Flow Sprinkler Space Capsule Car Acoustics Dosimetry Weathermen Wind Turbines Combustion Blood Flow ChinaCFD Gas Bubbles Side impact ColombiaBio

17 More teams in Round 2 Stent deployment Convection in a room Two-phase flow Wind turbines Remote Visualization Head acoustic field Drifting snow Ship hull flow Anchor bolt fastening Irrigation water sprinkler EMC and Dosimetry Weather simulation Radial and axial fan Gas turbine dilution Bicycle and rider flow Genomic Data Analysis

18 Next step: Production in the cloud The UberCloud, June 13, 2013

19 Team 2: Simulating new probe design for a medical device Front End + 2 GPU Solvers In Action

20 Team 2: Team members HPC Expert: Electromagnetic Simulation End User: Anonymous

21 Team 8: Multiphase flows within the cement and mineral industry Drying moist particles with hot gas

22 Team 8: Team members Ingo Seipp and Team End-User CFX

23 Team 26: Simulating Stent Deployment Using SIMULIA s Abaqus/Standard and Remote Viz Software from NICE to run CAE on SGI Cyclone Assessment of a fictitious balloon expandable stent design deployment, physiological pulsatile loading and both axial and radial compression.

24 Team 26- Team members End User: Anonymous: Global Designer and Manufacturer of Sterile Medical Products CAE Software Provider: Matt Dunbar, Chief Architect, SIMULIA Remote Viz Software: NICE Desktop Cloud Visualization (DCV) HPC/CAE Expert: Scott Shaw, Senior Applications Engineer, SGI Resource Provider: SGI Cyclone. Tony DeVarco, Senior Manager for Strategic Partners and Cloud Computing at SGI. Eugene Kremenetsky, Systems Engineering Technical Lead at SGI Team Mentor: Gregory Shirin from the HPC Experiment team

25 Team 26: What questions were being asked? 1. Is running Abaqus production jobs in a HPC cloud feasible? 2. How hard is it to set up cloud bursting capability when local resources are not available? 3. How well does remote visualization tools work? Open up a viz session Move Data; Submit; Run Job User Transfer Only Pixels to Visualize Transfer Data When Finished So what was to be tested by the End User in SGI Cyclone?

26 Team 26: Remote Visualization Interface

27 Team 26: Challenges, Barriers, Lessons Learned Getting customer s IT to open up firewall ports Data transfer via the network was found to be slow. Final results might be better transferred through an external USB hard drive via FedEx Data security and privacy assurance of the best possible data stewardship is critical from the standpoint of protecting the customer s intellectual property For an Abaqus user using SGI Cyclone is a viable solution for both compute and visualization The remote visualization aspect of the solution was impressive

28 Team 26: End User benefits 1. Gained increased understanding of using a cloud-based solution for doing Abaqus based FEA simulations 2. Shifting computational work to the cloud during periods of full utilization of in-house compute resources is a viable approach to ensuring analysis throughput 3. Participation in the experiment allowed direct assessment of speed and integrity of remote viz of computational models (both pre- and post-processing) for a variety of model and output database sizes 4. SGI/Nice DCV provides robust solution permitting fast and accurate remote manipulation of the computational models used in the study

29 Team 36: Combustion in an IC Engine

30 Team 36: Team members End User: Ferry Tap, Dacolt Combustion & CFD, The Netherlands, offers software and services for CFD modeling of industrial combustion applications CAE Software Provider: Wim Slagter, ANSYS HPC/CAE Expert: Ferry Tap: Dacolt Combustion & CFD Resource Provider: Matt Jacobs, Catherine Pringle, Penguin on Demand Team Mentor: Gregory Shirin from the HPC Experiment team

31 Team 36: Challenges of Diesel engine combustion Multi-physics: Evolving pressure and temperature conditions (compression / expansion) Liquid fuel injection, spray break-up, evaporation, mixing with air Auto-ignition, flame propagation and stabilization (chemistry) Interactions between turbulence, spray and chemistry Wall heat transfer and flame/wall interaction Pollutant formation Unsteady, two-phase, reacting CFD simulation with moving mesh

32 Team 36: Experiment findings CFD results A first impression on the performance of Dacolt PSR+PDF on an IC-engine case was achieved

33 Team 36: Challenges and Barriers Account set-up and end-user access Configuring end-user s CFD environment with Fluent Educating end-user in using batch queuing system Get data in and out of POD cloud Running end-user User Defined Functions with Fluent in parallel gave MPI problems Future use of HPC Cloud depends on availability of pay-as-you-go commercial CFD licenses

34 Team 41: Heavy Duty Abaqus Structural Analysis in the Cloud Realistic Simulation Requires HPC High-fidelity modeling Highly nonlinear materials Multi-physics Fracture and cracking Progressive damage and failure 34

35 Team 41: Team members and application The Team: Frank Ding, Simpson Strong-Tie; Matt Dunbar, SIMULIA; Steve Hebert, Nimbix; Sharan Kalwani, Intel The application: solving anchorage tensile capacity, steel and wood connector load capacity, and special moment frame cyclic pushover The Use Case: ABAQUS/Explicit and ABAQUS/Standard HPC cluster at Simpson Strong-Tie is modest, 32 cores Intel x86-based gear Cloud bursting is critical Sudden large data transfers are a challenge Need to perform visualization for ensuring design simulation is proceeding correctly

36 Team 41: Why HPC in the Cloud? Current HPC Cluster 4-node 32 cores total Nehalem-based Xeon InfiniBand DDR HPC in the Cloud Lack of in-house HPC resources Capacity surge large jobs speedup Capacity overflow jobs large number of jobs Project tested in Round 2 HPC Experiment Concrete anchor bolt tension capacity 1.9 millions of DOF s 11.5 hours runtime (32-core) 36

37 Team 41: Cloud-Based HPC Workflow Project Workflow Pre-processing on the local workstation at the user end Abaqus input file loaded to the data staging point thru SFTP Abaqus job submitted to the compute cloud thru Nimbix web portal Job monitoring thru Nimbix dashboard plus notification of job status Post-processing using remote visualization tool NICE DCV 37

38 Team 41: Barriers and Challenges Data movement limited by internet bandwidth Use remote visualization for post-processing End user side internet bandwidth issue Team member time schedule (voluntary effort) Team member expertise gap Meet the project deadline 38

39 Team 41: Benefits HPC in the Cloud solutions require multi-vendor support ISV 3DS SIMULIA provided Abaqus license Cloud infrastructure and service provider Nimbix HPC Cloud for CAE Remote visualization NICE Desktop Cloud Visualization(DCV) HPC Experiment provides a collaboration platform Form a team based on end-user application requirement Basecamp.com to support team communication Third-party solution providers invited to the team if needed 39

40 Bumps on the road Time delays: Vacation times in August and December No standards: Access and usage processes of hw & sw providers are different, some complex Hands-on: Process automation at providers greatly. Lack of automation: Currently the end-to-end process of the HPC experiment is manual (intentionally). Participants spent relatively small portion of their time, some are responsive, others are not: it is not their day job! Getting regular updates from Team Experts is a challenge because this is not their day job! vary

41 Are we discovering hurdles? Challenge: reaching out to industry end-users who are ready and willing to engage in HPC and especially HPC as a Service Half of our participants want to remain anonymous, for reasons: policies, internal processes, failure HPC is complex; at times it requires multiple experts Matching participants is critical to a team s success Resource providers (e.g. HPC Centers) often face internal policies, legal hurdles, licenses, Barriers: Complexity, IP, data transfer, software licenses, performance, specific system requirements, security, interoperability, cost, etc.

42 Why join? HPC as a Service is the next big thing, benefits are obvious HPC is complex; together it is easier to tackle the complexity Low entry barrier to HPC as a Service through an experiment Learning by doing; experimenting with no risk; no failure Becoming an active part of this growing community Exploring the end-to-end process and learning how this fits into your research and/or business direction in the near future

43 Conclusion HPC Experiment is a great experience for industry endusers bursting into HPC Clouds from their local system, and as part of their manufacturing workflow Response to the Uber-Cloud Experiment is overwhelming Everybody is learning and working along their very specific business or research interest In Rounds 1+2, 35 of the 60 teams finished successfully and in time, but effort has been underestimated by many 97% of the current participants will continue in Round 4 The experiment is helping the end-users to be more: agile, business flexible, profitable, cost effective,

44 Compendium of case studies 25 of our case studies are available for download:

45 Thank You