Cloud Computing for HPC

Transcription

1 Cloud Computing for HPC Extending Clusters to Clouds Solution Briefing TORONTO 10/25/2011

2 Company Background 2

3 Platform Computing, Inc. Platform Clusters, Grids, Clouds Computing The leader in cluster, grid and cloud management software: o 19 years of profitable growth o 2,000 of the world s most demanding client organizations o 5,000,000 CPUs under management o 500 professionals working across 13 global centers o Strategic relationships with Cray, Dell, Fujitsu, HP, IBM, Intel, Microsoft, Red Hat and SAS 3

4 Product Leadership We believe Platform ISF is perhaps the most complete internal cloud software solution we ve seen so far, Staten says. Platform HPC Platform MPI Platform LSF Platform Symphony Platform ISF Clusters Grids Clouds Workload Management Resource Management Platform Computing 4

5 Cloud Computing for HPC 5

6 HPC Market Trends Key Trends Extending HPC to the cloud paradigm Management, Analysis, and Reporting Graphic Processing Units ISV Application Integration 6

7 HPC Challenges USERS IT Users need: More grid resources to run application faster Flexible resources to support multiple application Lower cost per performance IT needs: Contain costs without compromising grid size and performance Grid data security Better meet their users needs 7

8 Cloud Computing Can Help Optimally Making Use Of The Cloud Send workload to the cloud o When workload queues exceed tolerable thresholds for pending jobs o When more capacity is required to meet SLAs o To help small groups easily run their first cluster enabled jobs Not all workload is suitable for the cloud, including: o When data transfer required exceeds the acceptable wait time o Data intensive computing applications o When cloud resources become unreliable or unavailable o When privacy and/or security risks are too high 8

9 HPC Cloud - Differentiating the products Cloud Connect On Demand LSF, HPC, Symphony plugin Schedules infrastructure in multitenant environments Provides for request fulfillment for sandboxing Creates external cloud connection from local clusters Accounts for infrastructure consumed Customer self service Pay-per-use Amazon EC2 only (no local) Web access only HPC Vertical specific (Life Sciences offering now avail) Applications pre-installed Platform HPC only 9

10 10

11 Problem #1: Connecting to the cloud Problem Customer workloads are spiky Provisioning for peak is highly wasteful (utilization) Relying on desktops or existing servers wastes user time and can be very slow Alternatives Take advantage of cloud resources by building their own solution Provision for peak (and live with the cost) Wait (wastes valuable engineering time, slow TTM) Desired solution Provide a simple to use IaaS connection from an LSF Cluster Provide a simple policy engine to decide which jobs burst and which wait for local resources 11

12 Problem #2: Only use what is needed Problem IaaS providers usually charge by the instance-hour. In short bursts, very cost effective. In long duration, expensive. Workload varies all the time. Cloud should only be used for peak demand Alternatives Open Source products: OpenNebula, Nimbus Competitive products: AdaptiveCloud & Unicloud In-house ELIM integrated with IaaS APIs Desired solution LSF/HPC/Symphony Plugin architecture Automated flexup/flexdown based on pending jobs and TTL for idle resources 12

13 Platform s Product Line Products that make up Platform s cloud solutions Cloud Extension HPC Appl. Integrations Advanced Analytics Test/Dev Private Cloud Workstation Cluster Extension HPC LSF Symphony ISF Java PCM MPI EGO GPU GPU GPU 13

14 What is Platform HPC? The easiest and most complete HPC cluster solution Complete Product Integrated Application Support Certified Feature-rich workload management Unified web portal for access anywhere Heterogeneous cluster management Easy to use job submission portal Customizable application templates Certified with server, storage & interconnect vendors Best customer support Platform HPC 14

15 What is Platform LSF? The HPC Workload Management Standard Complete Advanced, feature-rich workload scheduling Robust set of add-on features Integrated application support Powerful Policy & resource-aware scheduling Resource consolidation for max performance Advanced self-management Most Scalable Support for thousands of concurrent users & jobs Delivers a virtualized pool of shared resources to support multiple apps Flexible control to support multiple policy centers Best TCO Optimal utilization reduces infrastructure costs Improves user productivity for faster time to solution Robust operational capabilities improve administrative productivity 15

16 What is Platform ISF? Dynamic Resource Management Separate applications from infrastructure by creating an independent management platform... Application workloads Private Cloud management platform IaaS VM management Provisioning Server Storage network... to achieve resource sharing, vendor independence, and commodity computing 16

17 Platform s Cloud Solutions for HPC Cloud Bursting Making it easy to extend to the cloud 1 Integrated Cluster with the Cloud (Platform LSF with VPN cluster management) 2 Multi-Cluster to the Cloud (Platform LSF with Platform MultiCluster) 3 Dynamic Cluster Extension to the Cloud (Platform LSF with Platform ISF) 17

18 Use Case #1 Integrated Cluster with the Cloud Workload Manager (Platform LSF) A policy determines that the jobs can go to the cloud 4 Jobs 3 Internal Resources Platform LSF contacts cloud provider to launch VMs 5 Cloud provider VPC connection User 2 End user submits more jobs 1 The existing cluster nodes are already too busy 6 Additional resources from Amazon join automatically an existing cluster 18

19 Use Case #2 Multi-Cluster to the Cloud Workload Manager (Platform LSF) 3 Transparent for end users User 2 Jobs Internal Resources 1 4 LSF asks cloud provider to create a new cluster of VMs (possibly CCI) Cloud on-demand instances Jobs that may go to the cloud in RED MCO automatically forwards jobs to the new cluster based upon poliicies MultiCluster orchestrator 5 19

20 Use Case #3 Dynamic Cluster Extension to the Cloud User 1 I need a cluster: 1 master and 3 computes nodes to test my new project Jobs Workload manager requests resources from dynamic resource manager 2 3 ISF determines that no internal resources are available and by policy QA/TST can go to cloud Workload Manager (Platform LSF) 7 6 ISF passes master location of new nodes to LSF Dynamic Resource Manager (Platform ISF) ISF gets master location from cloud provider 5 Cloud provider gets request from ISF to build new test cluster 4 Grid is created and then jobs get submitted 20

21 Platform s Cloud for HPC Key Benefits Designed for Extensibility Customizable scheduling algorithm Open adapter architecture Flexible architectural options On Demand Scalability Grow the grid when needed, shrink when not Contain capital costs, keep utilization high Industry-Leading Support Large, worldwide development and support team Extensive partner ecosystem Nearly two decades of HPC experience 21

22 22

23 Problem #1: No HPC Infrastructure locally Problem Smallest companies and consultants have no access to HPC Provisioning for HPC need is unwise / impractical Desktop is insufficient to service the workload Little or no IT/HPC expertise Alternatives Very few Contract with Cycle Computing, Univa, others to build cloud infrastructure (expensive, long lead time) Desired solution Self service, near instantaneous availability, security Provide pre-configured SaaS (open source apps) for anyone with an IaaS account Applications pre-installed, pre-configured, ready to execute 23

24 Platform OnDemand Workstation User VPN OR MultiCluster L S F D A T A F I L E D A T A Phase I: Life Sciences Phase IB: GRE 24

25 Roadmap On Demand HPC Cluster Linux Win Joined Indep L/S & Chem n n n n CAE & IM O/G GEO n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n GRE DCC FS EDA Phase I Phase IB Phase II Phase III Phase IIIB Phase IV Phase I IB Phase II Phase III IIIB Phase IV CYQ CYQ CYQ CYQ Independent Offering Marketplace GA 25

26 EC2 Instance Sizes Vertical Master Host HPC Cluster Life Sci. / Chem Standard Large HPC CAE / IM Standard Large HPC Oil & Gas / GEO Standard Large HPC GRE Standard Small High-Mem 4 XL DCC TBD TBD EC2 Instance Options Size Memory (GB) Cores HDD (GB) Price $(Lin / Win) Standard-Small / 0.12 Standard-Large / 0.48 Standard-XL / 0.96 Micro EBS only 0.02 / 0.03 High-Mem XL / 0.62 High-Mem 2XL / 1.24 High-Mem 4XL / 2.48 High CPU-med / 0.29 High CPU-XL / 1.16 HPC / - 26

27 Thank You 27