Bright Cluster Manager

Transcription

1 Bright Cluster Manager A Unified Management Solution for HPC and Hadoop Martijn de Vries CTO

2 Introduction

3 Architecture Bright Cluster CMDaemon Cluster Management GUI Cluster Management Shell SOAP/ JSONAPI +SSL SOAP+SSL node001 Web-Based User Portal head node node002 Third-Party Applications node003

4 Management Interface Graphical User Interface (GUI) Offers administrator full cluster control Standalone desktop application Manages multiple clusters simultaneously Runs natively on Linux, Windows and OS X Cluster Management GUI Cluster Management Shell (CMSH) All GUI functionality also available through Cluster Management Shell Interactive and scriptable in batch mode Cluster Management Shell

5

6 Hadoop Integration

7 Managing Clusters Bright Cluster Manager can be used for several types of clusters HPC Compute Storage Private cloud (OpenStack) Server farms Big Data (Hadoop) All types of clusters need to be: Deployed Configured Provisioned Managed Monitored Health-checked

8 Managing Hadoop Clusters Managing Hadoop Clusters just as difficult as other types of clusters Without proper infrastructure, Hadoop will not run and cluster will not be usable for data processing Bright Cluster Manager provides single-pane-of-glass to manage and monitor all aspects of Hadoop cluster Includes: Hardware (set up, configuration, monitoring) Operating system (provisioning, updates) Hadoop distribution Hadoop configuration Users Bright Cluster Manager provides perfect environment for Hadoop to run on Hadoop distribution agnostic (switching is easy)

9 Bright for Hadoop Cluster Management Bright Cluster Manager 7.0 for Apache Hadoop Provides single-pane-of-glass for managing both physical cluster as well as Hadoop Easy installation of Hadoop Apache Hadoop 1.2, 2.2, 2.3 & 2.4 (on Bright DVD) Cloudera CDH 4 & 5 HortonWorks HDP 1.3 & 2.1 Configuration, monitoring and healthchecking of Hadoop instances Graphical UI, command-line interface and API access

10 10

11 11

12 12

13 13

14 Hadoop Configuration Hadoop configuration through roles Nodes can be configured to run certain Hadoop related services by assigning roles Example roles: DataNode, JobTracker, TaskTracker, Namenode, SecondaryNameNode, YARNServer, YARNClient, HBaseServer, HBaseClient, ZooKeeper Assigning/unassigning role will: Write out configuration files based on role parameters Start/stop/monitor relevant services Most important Hadoop configuration aspects can be changed from inside Bright Exotic Hadoop configuration parameters can be set directly in (partially generated) configuration file

15 15

16 Hadoop Management Features Integrated user management and HDFS access control Ability to re-purpose nodes between Hadoop and e.g. HPC Multiple HDFS instances on same cluster (different Hadoop distributions possible) Most Hadoop configuration aspects controlled through GUI and CLI Healthchecking and monitoring of Hadoop related services Ability to use alternative filesystems to HDFS (e.g. Lustre)

17 Re-purposing nodes Node tasks are determined by assignment of roles (e.g. Hadoop Data Node, Slurm) By default, node runs all tasks that it has been assigned roles for in parallel (e.g. Hadoop + Slurm) Two methods to stop running Hadoop on a node: Method 1: (temporary) Property at category and device level: Use exclusively for: Values: <empty>, HPC, OpenStack, Hadoop, Ceph, Nothing Setting Use exclusively for causes all other tasks to be stopped immediately Method 2: (permanent) Hadoop related operations: decommission/recommission Decommission: move data to other nodes to maintain replication factor and stop using for jobs (could take a while) Recommission: move data back to node and use for Hadoop jobs

18 Conclusion Bright provides tried & tested method of cluster management Hundreds of clusters world-wide are being managed using Bright Cluster Manager Inclusion of Hadoop management capabilities provides complete solution for setup, management & monitoring of Hadoop clusters Single pane of glass for cluster & Hadoop Especially well suited for clusters that must support both HPC compute and Hadoop jobs

19 Questions?