Hybrid Cluster Management: Reducing Stress, increasing productivity and preparing for the future

Transcription

1 Hybrid Cluster Management: Reducing Stress, increasing productivity and preparing for the future Clement Lau, Ph. D. Sales Director, APJ Bright Computing

2 Agenda 1.Reduce 2.IncRease 3.PrepaRe

3 Reduce System Administrator? 3

4 Reduce

5 The Problem According to IDC Clusters are still hard to setup, operate, use and manage New buyers require ease-ofeverything The shortage of skilled people is a major hindrance Software is still the #1 roadblock Better management software is needed 5

6 The Problem clusters are difficult All clusters may need deployment provisioning updating managing monitoring security health checking alerting/alarming virtualization cloud workload management What makes it hard everything affects everything else everything changes all the time hardware breaks all the time cloud, virtualization increase complexity IT Pain Downtime Delays Personnel cost Inability to react 6

7 Managing hybrid clusters more complex Artist: mmpninja 7

8 Reduce Some Causes of Stress Many users with different system requests want different CUDA SDK versions High stress levels when there is a system problem and the users are reporting it High stress levels trying to identify and fix the problem(s) Experienced HPC analysts and System administrators spending too much time to manage the cluster instead of doing more productive work Low job satisfaction related to cluster management

9 Reducing Stress how can we help? Provide intuitive, single pane of glass solution to cluster management Provide robust, integrated and customizable system monitoring and alerts so that system administrators are first to know of problems

10 Bright Cluster Manager - Single Pane of Glass 10

11 HPC Accelerators NVIDIA GPUs

12 GPU Management Software Everything needed to enable GPUs on a cluster is packaged as easy-to-install Bright packages: NVIDIA drivers CUDA & OpenCL Environment modules ensure Perfect setup of user environment (PATH, LD_LIBRARY_PATH,...) Easy switching between CUDA versions CUDA driver recompiled automatically against running kernel at boot-time Management Set-up wizard takes care of initial GPU configuration (e.g. creating bridge interfaces, assigning IP addresses) GPU appears as a first-class device type in cluster management infrastructure GPU can be configured, controlled and monitored through CMSH and CMGUI Support for multi-gpu units WorkLoad Management GPU is automatically added to the workload management system as a consumable resource Compute jobs may request GPU resource in job script Monitoring & Health Many GPU metrics monitored Many GPU health checks configured

13 Bright s GPU specific features Configuration options for GPUs: ECC mode Compute mode Operation mode Persistence mode Power limit Clock speeds Two additional nvidia-healthmon based health checks to verify that GPU is working properly: Quick check which can be run periodically Extensive check which should not be run concurrently with jobs

14 1 ABOUT

15 About Bright Who we are Enterprise infrastructure software company Founded in 2009 Bare-metal & cloud provisioning, monitoring, management of clustered systems in the data center: Hadoop, private cloud, HPC, storage, database and other clustered systems Number of servers Supercomputer Hadoop cluster OpenStack private cloud Unique because fully integrated and therefore easier to use, more efficient, more powerful, more comprehensive Backed by DFJ, DFJ Esprit, Prime Ventures, ING CIP Our success Customers around the world: Boeing, ING Bank, Lincoln Financial, Novartis, NASA, DoD, DoE, Stanford, Oracle, HP, Intel, Sinopec and >400 more Resellers around the world: OEMs: Acer, Atos/Bull, Cisco, Cray, DDN, Dell, Huawei, Lenovo Integrators: >100 around the world Headquarters Amsterdam, the Netherlands San Jose, USA Award winning

16 Bright Cluster Manager - Background Over 10 years in development Used by 400+ customers (Commercial and Academic) worldwide Single pane of glass provisioning, management, monitoring and alerts Vendor agnostic supports many x86 system vendors Multi-Cluster manage multiple clusters in multiple geographic locations through the same single interface Deploy, manage and monitor accelerators: GPGPU and Xeon PHI (MIC) Scalability up to 10,000 nodes today growing to Exascale

17 Bright Cluster Manager : The Integrated and Comprehensive Deployment Provisioning Dashboards Monitoring Visualization Health checking Management Workload management Security Application specific HPC Hadoop Private cloud Cloud extension 17

18 The Comprehensive Solution Deployment Provisioning Dashboards Monitoring Visualization Health checking Management Workload management Security Application specific HPC Hadoop Private cloud Cloud extension 18

19 19

20 20

21 Result Stress Reduced! 21

22 Reduce More 22

23 Bright Cluster Manager: Helping to save power Bright Cluster Manager s cm-scale-cluster utility can help save power by: Changing the power state of compute nodes Saving Energy and storage costs Changing the existence state of compute nodes depending on workload demands Stops or terminates compute nodes when no queued or running jobs remain on the managed nodes 23

24 Agenda 1.Reduce 2.IncRease 3.PrepaRe

25 Increase

26 Increasing ROI ROI = Faster research results Faster time to market More profits Some factors that help increase your ROI: Reducing time it takes to deploy a cluster Reducing the amount of effort required to monitor and maintain a cluster Reducing down time (Increasing Uptime)

27 Bright Cluster Manager Increase Uptime Bright Cluster Manager s rapid rebooting across nodes has reduced our node provisioning by a factor of ten, and it s done with a few mouse-clicks, Greg Siekas, High Performance Computing Technical Lead at Boeing. Source: InsideHPC

28 Image Management: Guarantee Job Execution Environment Deployment Provisioning Dashboards Monitoring Visualization Health checking Management Workload management Security Application specific HPC Hadoop Private cloud Cloud extension 28

29 Increasing Efficiency

30 Black Hole Node Syndrome The black hole node syndrome is a serious issue. There are many subtle problems from apparently healthy nodes that can create cascading job failures. And the bigger the job, the more nodes, the higher the probability of failure. Dr. Don Holmgren, Computer Services Architect at Fermilab Source: HPC Wire

31 Avoiding Black Hole Node Syndrome Bright Cluster Manager includes a framework that enables system administrators to define pre-job health check: specific low-impact tests that are run just before a job is executed to identify illnesses that can affect jobs. Bright instructs the workload manager to hold the job briefly while the nodes reserved for this job and other system elements are tested. If any node fails the health check, predefined actions are executed. Bright and the workload manager then dynamically reschedule the job to a set of healthy nodes while alerting the system administrator. The cluster remains productive, and the unhealthy node is flagged for further attention.

32 Bright Cluster Manager IncReasing ROI by: Reducing time it takes to deploy a cluster Reducing the amount of effort required to monitor and maintain a cluster Reducing down time (Increasing Uptime)

33 Agenda 1.Reduce 2.IncRease 3.PrepaRe

34 Change is inevitable

35 Agenda 1.Reduce 2.IncRease 3.PrepaRe

36 Bright Cluster Manager helping prepare for the future Two largest growth areas in HPC: Big Data Hadoop Private clouds using Openstack

37 Hadoop with Bright Cluster Manager Easy installation of Hadoop on top of Bright cluster Installs from bare metal Same Bright Cluster interface to deploy, manage and monitor Hadoop instances Support for several Hadoop distributions CDH4 (Cloudera) HDP (Hortonworks) Intel Hadoop Vanilla Hadoop 1.2 and 2.0 (supplied on Bright DVD) Pivotal HD Intel Enterprise Edition Lustre (IEEL) with Hadoop Hadoop specific monitoring metrics Integration with Hadoop Workload Managers Automated installation (not involving e.g. Cloudera Manager, Ambari)

38 38

39 Bright Cluster Manager Summary: Architecture AWS, Azure, Google, OpenStack HPC Big Data Othe r Private Cloud Bright Cluster Manager Bright Bright Linux Linux Linux Linux Linux Linux Linux Linux Win Deb VM VM VM VM VM VM VM VM VM VM HPC Big Data Other Private Cloud Bright Cluster Manager & Bright OpenStack Linux Linux Linux Linux Linux Linux Linux Linux Server Server Server Server Server Server Server Server

40 Conclusion Bright Cluster Manager provides the 3 R s of Cluster Management 1.Reduce 2.IncRease 3.PrepaRe

41 Thank You! For more information and discussion: Recommended Reading: InsideHPC Whitepaper: Strategies for Managing High Performance GPU clusters