GPFS Management und Monitoring mit dem IBM Platform Cluster Manager (PCM) Standard Edition

Transcription

1 GPFS Management und Monitoring mit dem IBM Platform Cluster Manager (PCM) Standard Edition 1

2 Agenda Platform Cluster Manager SE Wo? Wann? Warum? Im Detail Architektur und Support PCM und GPFS Deployment: Kits Deployment: Packages Monitoring Demo Falls die Zeit reicht Fazit 2

3 IBM Platform Cluster Manager WO? WANN? WARUM? 3

4 IBM Platform LSF 9 Family IBM Platform Analytics IBM Platform Application Center IBM Platform RTM IBM Platform Dynamic Cluster IBM Platform LSF Standard Edition Includes MultiCluster, Make & Client IBM Platform Session Scheduler IBM Platform Process Manager IBM Platform License Scheduler 4

5 IBM Platform HPC Easy to use, complete and integrated HPC cluster management software Platform Cluster Manager Platform LSF + GPU Scheduling Platform Application Center Platform HPC Platform MPI Integration 5

6 HPC Cluster Architecture Public network Private network typically Ethernet, compute network, management, provisioning etc. Management node / Head node Storage High speed switch High speed interconnects typically 10 GE or InfiniBand 6

7 Unique Capabilities Operating system(os) provisioning 7 It supports Red Hat Enterprise Linux family operating systems for both x86-64 and POWER systems, and Suse Linux Enterprise Server. It can provision the operating system together with software components Kits framework simplifies software deployment and maintenance in a technical computing environment This framework is used within an ecosystem to deploy software like InfiniBand drivers, GPU runtime software, and so on Robust and scalable monitoring system that is easy to extend and customize The monitoring capability expands the entire infrastructure that includes server, networking, and storage. The monitoring agent uses the same technology as is used in IBM Platform LSF and IBM Platform Symphony offerings Easy to use, centralized web portal that allows administrators to access and manage the cluster from anywhere Reports on statistics of the cluster usage and performance. Greater management scalability via embedded xcat technology. PCM SE v4.1.1 can scale up to 2,500 nodes via GUI. 3x increased scalability from its previous version. Management service automated failover

8 Benefits Faster time-to-cluster readiness Reduced infrastructure and management costs Improved administrator productivity Increased cluster uptime Greater management scalability 8

9 9

10 IBM Platform Cluster Manager im Detail 10

11 Comparison of xcat, Platform Cluster Manager Standard Edition & Platform HPC xcat Platform Cluster Manager-Standard Edition Platform HPC Linux x64 support AIX support Windows support Linux for Power support Scalability Hardware management Hardware and system monitoring Management portal One step install Ease of use (provision template etc.) 3 rd party software kit (ICR, OFED etc.) Management service HA Workload management Platform MPI Job management portal Open source Commercial support 10,000+ 2, Requires 3 rd party monitoring agent optional 11

12 IBM Platform Cluster Manager Standard Editions vs. Advanced Edition Platform Cluster Manager Standard Edition Software for managing a static computing cluster for a group of users Platform Cluster Manager Advanced Edition Software for managing dynamic clusters in a multitenant environment Single User Group No No Upgrade upgrade Path. path Can t Cannot co-exist Coexist Multiple Tenants Private Network Private Network PCM SE Computing Cluster PCM AE Resource Pools 12

13 Intuitive Web-based Interface Objects tree allows easy navigation Alerts summary Cluster health Hide the complexity of a cluster Easy to learn and easy to use Cluster performanc e 13

14 Provisioning templates GPFS Experten Workshop Mainz, Provisioning Templates Provisioning template simplifies deployment of nodes for specific purpose HPC SLES + OFED + MPI+GPFS CentOS + GPFS Image profile RHEL Big data Network profile 1 Network profile 2 Network profile Embarrassingly parallel IBM Pure server IBM idataplex server Other x86 server Hardware profile 14

15 Managing Provisioning Templates & Profiles Provisioning templates, image profile and network profiles can be easily managed in GUI 15

16 Rackview in Dashboard Rackview provides a view of physical layout for the cluster It allows administrator to quickly examine the status of individual nodes Admin can drill into node view by clicking the node Chassis console can be launched from the rack view 16

17 Node Monitoring & Management Node detail is monitored Management capabilities include: power cycling, firmware updates, OS reboot, reprovision, synchronize, control LED on the node, launch BMC console, launch ssh or VNC console etc. Node list can be sorted by different category The monitored metrics, performance charts, alerts are organized in tabs 17

18 Alerts Highlight Potential Issues in The Cluster Alerts can be defined using any monitored metrics Alert can trigger an automated action with custom script Alert history shows the detail of the triggered alert 18

19 Reporting Historical reports can be generated for Cluster availability Cluster performance and usage Free application licenses Custom reports can also be created using the data collected by PCM 19

20 ARCHITEKTUR UND SUPPORT 20

21 Provisioning and Management Architecture Web Portal Frontend: web portal xcat Node Mgt Node Update (CFM) Node HW Mgt Backend: xcat is provisioning as management foundation Features coded as xcat Plugins Rack/ Chassis Mgt Image Mgt OS Distro Mgt Kit Mgt OS Distro Update Mgt Network Mgt Data: xcat Plugins store cluster configuration inside PostgreSQL DB PostgreSQL Database 21

22 Monitoring Architecture 1) Collect Data 2) Store Data 3) Display Data Monitoring (PERF) Monitoring (PERF) Web Portal xcat EGO PostgreSQL Database PostgreSQL Database 22

23 PCM-SE Supported Environment Hardware: IBM System x rack servers IBM idataplex servers IBM Flex System servers x and Power IBM Power Linux rack servers Generic x86 servers from other vendors Supported OS Node Supported x86 OS Supported Linux for Power Management Node - RHEL x64, CentOS x64 - SLES 11 SP2, SP3 - RHEL for Power Compute Node (Stateful) Compute Node (Stateless) - RHEL 5.8, 5.9 x64, CentOS 5.8, 5.9 x64 - RHEL x64, CentOS x64 - SLES 11 SP2, SP3 - RHEL for Scientific Computing* - RHEL 5.8, 5.9 x64, CentOS 5.8, 5.9 x64 - RHEL x64, CentOS x64 - SLES 11 SP2, SP3 - RHEL for Power - RHEL for Power Requires patch from Fix Central. We tested RHEL 6.5 after was released 23

24 PCM und GPFS - Deployment: Kits 24

25 25

26 Kits PCM kits allows software deployment without writing scripts A kit includes multiple components Kits can be added to an image profile together with the OS 26

27 IBM HPC Software Kits New in xcat 2.8. Supported for Linux OS images only.xcat Kit support replaces the Linux IBM HPC Integration Support (xcat-ibmhpc) previously shipped with xcat 2.7.x and older releases. IBM product software kits combine the product packages with configuration files, installation scripts, environment variables, exclude lists, and other data that is unique to deploying and running that product in an xcat cluster. Kit components for different cluster roles such as compute, storage, login, utility, servicenode, and mgtnode are available as supported by the product. Currently, there are only partial product kits available for most products. Complete kits for product software will be shipped with the next new release of the product on the product distribution media. A partial kit is a tar file that contains everything except for the actual product packages. Instructions are provided below for how to complete a partial kit by adding all of the product packages and rebuilding the kit tar file. Only complete kits can be added to an xcat cluster. Source: 27

28 Obtaining partial IBM HPC Software Kit for GPFS The partial product kits are avalable from the FixCentral download site. To download a kit: Select the Product Group "Cluster Software" <continue> Select General Parallel File System" <continue> Select Installed Version, for example "All" <continue> Select Platform, for example "All" <continue> Identify Fixes, select "Browse for fixes" <continue> Select Fixes, select the product kit that most closely matches the release, version, and architecture for your product. <continue> Note that not every fixpack for a product may have a corresponding new partial kit, so you may need to find the last kit fixpack that was previously released. If prompted, sign into FixCentral with your IBM Id and Password <Sign in> Review and accept "Terms and Conditions" popup Select the kit tar file to download Source: 28

29 Download des GPFS xcat KIT 29

30 Erstellen des GPFS Kits aus dem partial Kit Completing the kit build for a partial kit Typically, a software kit will contain all of the product package files. However, in some instances, software kits may be delivered as partial or incomplete kits, and will not include all of the product rpms. You will then need to obtain your product packages and license through your reqular distribution channels, download them to a server along with the incomplete kit, and run the buildkit addpkgs command to build a complete kit that can be used by xcat. Note: The software license package will be available from the product media. If you download product software it will not include the license package. When completing a partial KiT you must make sure to include the license package when you "addpkgs" to the partial Kit. The file name of the kit tarfile will indicate if it is a partial kit. For example, if the name of your product kit tarfile is something like: product-version-linux.need_product_pkgs.tar.bz2 the string "NEED_PRODUCT_PACKAGES" in the file name indicates that you have a partial kit. If you received your IBM HPC product software kit as a partial kit, follow these steps to complete the kit build process: Install the optional xcat-buildkit rpm on your server. This rpm will not automatically install with your other xcat packages and does not necessarily need to be installed on an xcat management node. The xcat-buildkit rpm does not have any dependencies on any other xcat packages. However, it does require the rpmbuild and createrepo commands to be available. For RHELS, these commands are provided by the rpm-build and createrepo packages respectively. For SLES, they are provided by the rpm and createrepo packages(note that createrepo is shipped on the SLES SDK iso). Download the kit tarfile and the product package files and make them locally available on your server. cd to a work directory Build the complete kit tarfile: buildkit addpkgs <kit.need_product_pkgs.tar.bz2> --pkgdir <product package directory> You have to create the rpm for the portability layer as well and put it into the product package directory (cd /usr/lpp/mmfs/src; make Autoconfig; make World; make InstallImages; make rpm) Source: 30

31 GPFS base level version und Kit-Level müssen übereinstimmen, Sonst schlägt buildkit fehl. 31

32 PCM und GPFS - Deployment: Packages 32

33 Kopieren der GPFS rpms nach /install/contrib/sless11.3/x86_64 Und starten des imageprofileloaders, damit die RPMs im GUI als custom package sichtbar werden. NUR Base level rpms hierhin packen, Da es mit Upgrade rpms im gleichen Ordner Probleme bei der Installation gibt (GPFS will zuerst den Base Level installiert bekommen, der Installer findet aber den Upgrade und will nur den installieren. 33

34 Auswahl eines PCM Image Profiles a.k.a. xcat osimage. Dann Modify button drücken 34

35 Auswahl der GPFS rpms 35

36 Automatisches Update (Installation der RPMs) aller Knoten, die mit dem Image Profile (osimage) installiert wurden. 36

37 Synchronisations-Status (failed, weil die KVMs nicht gebootet waren...) 37

38 PCM und GPFS - Monitoring 38

39 GPFS Storage Monitoring By default, IBM General Parallel File System (GPFS ) storage monitoring is NOT installed. You must first set up and configure GPFS storage monitoring to use it in the Web Portal.Before you begin GPFS storage monitoring enablement has the following restrictions: GPFS storage monitoring is supported for GPFS Only one GPFS cluster can be enabled for GPFS storage monitoring in Platform Cluster Manager. GPFS storage monitoring does not support a GPFS cluster with high availability. Procedure Log in to the management node as a root user. Setup root passwordless SSH access to the GPFS quorum-manager node on the management node. The GPFS RPM (pcm-gpfs-monitoring x86_64.rpm) is in the /install/kits/kit-pcm /repos/kit-pcm rhels-6-x86_64 directory. From this directory, run the following command to install the GPFS monitoring package onto the management node: # rpm -ivh pcm-gpfs-monitoring x86_64.rpm Configure the GPFS quorum-manager node IP address or fully qualified domain name (FQDN) on the management node, where gpfs-quorum-manager is the GPFS quorum-manager node hostname. # pcmgpfstool -c -n gpfs-quorum-manager-node-name Restart data collection monitoring in Platform Cluster Manager. # perfadmin stop all # perfadmin start To view your GPFS storage, login to the Web Portal. You can view your GPFS storage in the Resources tab under the Storage (GPFS) option.parent topic: GPFS storage Source: 39

40 Installation und Aktivierung des GPFS monitorings. Danach war ein Restart des PCM erforderlich, um die Anzeige im GUI zu aktivieren. 40

41 PCM GPFS loader laden regelmässig (Polling!) Informationen in die PCM Datenbank, die dann vom WebGUI angezeigt werden (Zeitversatz!). Mit plcclient.sh d <loader> kann man das manuell triggern! plcclient.sh s zeigt die Liste der Loader und das Polling Intervall in Sekunden. 41

42 GPFS Cluster Summary 42

43 GPFS Cluster Knoten 43

44 GPFS File System 44

45 GPFS NSDs 45

46 Fazit 46

47 Fazit Platform Cluster Manager SE ist stark in Einfache und schnelle Installation von HPC Clustern Intuitives, graphisches Management über Web-GUI Monitoring des Zustands des Gesamtsystems (Knoten, Switches, GPFS) Dafür müssen Zugeständnisse gemacht werden Einfache Installation ohne grosses Customizing (postscripts) klappt bei der vorgegebenen Default-Architektur und wenn man sich an die Vorgaben hält Kein supportetes Command Line Interface (heute) Das Out of the box Monitoring ist begrenzt und zeigt nicht immer alles, was man sehen möchte Der Ansatz kann für viele Kunden, gerade mit geringem HPC Skill und einfachen Clustern, sehr gut funktionieren. Kunden mit sehr gutem HPC und xcat Skill und existierenden Umgebungen werden möglicherweise andere Tools bevorzugen. 47

48 Thank you! For links see the backup slides 48

49 Wichtige Links IBM Platform HPC and Cluster Manager Wiki M%20Platform%20HPC%20Wiki/page/Welcome%20to%20the%20IBM%20Plat form%20hpc%20and%20cluster%20manager%20wiki PCM Installation Guide PCM Administration Guide PCM Release Notes 49