HiPerDNO. High Performance Computing Technologies for Smart Distribution Network Operation FP

Transcription

1 HiPerDNO High Performance Computing Technologies for Smart Distribution Network Operation FP Project coordinator: Dr Gareth Taylor (BU) Consortium Members: Brunel University (BU), Electricite de France (EF), IBM Israel, University of Oxford, UK Power Networks plc (UKPN), Union Fenosa Distribution (Union Fenosa), Indra Sistemas, GTD Systems de Information (GTD), Korona Inzeniting (Korona), Elektro Gorenjska Podjetje za Distribucijo Elektricne Energije (EG) and Fraunhofer IWES Document Title Prototype deployment of selected architecture and necessary documentation, D1.2.2 Document Identifier HiPerDNO/2011/D Version Version 1.0 Work package number Sub-Work package Number Distribution Reporting consortium member WP1 WP1.2 Public Oxford Internal reviewer & review date David Wallom (July 2011) FP

2 Prototype deployment of selected architecture and necessary documentation, D1.2.2 Executive Summary This document presents the HiPerDNO HPC Platform prototype developed in WP1.2. The prototype covers all major components of the HiPerDNO HPC platform, and it was presented during the HiPerDNO Quarterly Progress Meeting in Clamart on 29 th 30 th June The prototype can be downloaded from the FTP server hosted by Brunel University. The prototype can be installed and run following the instruction and tutorial in this document. This document also lists the requirements for the HPC scheduler. These requirements are met by the HPC prototype, and represent our current understanding, formed through interaction with the HiPerDNO partners, of the needs of DNO applications. Obviously, as collaborations with partners progress so will the specification and deployment of their needs within the HiPerDNO HPC Platform. FP

3 Document Information HiPerDNO Project Number Full Title FP Acronym HiPerDNO Prototype deployment of selected architecture and necessary documentation, D1.2.2 Project URL Document URL N/A Deliverable Number D Title Prototype deployment of selected architecture and necessary documentation (M18) N/A Work Number Package WP1.2 Title Investigation and evaluation of new HPC platforms and architectures. Date of Delivery Work Plan Actual Status Version 1.0 Nature Prototype Internal Report External Report Dissemination Dissemination Level Public Consortium Author(s) (Partners) Lead Author Abstract Stef Salvini, David Wallom Name Piotr Lopatka Partner Oxford Phone This document and the prototype of the HPC System constitute deliverable D The prototype covers all major components of the HiPerDNO HPC platform, and it was presented during the HiPerDNO Quarterly Progress Meeting in Clamart on 29 th 30 th June The prototype can be downloaded from the FTP server hosted by Brunel University. Access details can be found on Basecamp. This document describes the prototype, its installation and use. Keywords prototype, virtualization, HPC, Torque, MAUI, HDFS, NFS FP

4 Table of Contents 1. Introduction Prototype Description... 6 Purpose of the Prototype... 6 Functional Description of HPC Prototype... 6 HPC Interface to the Data System... 7 Applications scheduling Prototype Deployment Installation Tutorial... 9 Prerequisites... 9 Creation of VM... 9 Virtual network setup Head-Node: hpc Compute-Nodes: hpc1, hpc2, hpc3, hpc Fast Storage node: storage Data System nodes: hdfs1, hdfs Testing the Prototype DNO-like model applications Running the example Other key commands (HDFS, Torque, MAUI) Editing schedules From Prototype to System Deployment References FP

5 1. Introduction This document describes the prototype of the HiPerDNO HPC Platform that was described and demonstrated at the HiPerDNO Quarterly Progress Meeting in Clamart on 29 th -30 th June The prototype implemented the design concepts introduced in the Deliverable The chapters of this document discuss the prototype implementation, deployment and the tests that were conducted. The HPC prototype software package has been made available to all partners through the FTP server hosted by Brunel University. Installation instructions are included in this deliverable. FP

6 2. Prototype Description Essential reading : HiPerDNO deliverables [1], (particularly chapters 2 and 3) [7]. The prototype consists of the HPC Engine (HPCE) and the Data Subsystem (DSS). The HPCE schedules and executes the DNO applications making optimal usage of resources while ensuring appropriate balance and priority between mission critical and non-critical applications. The Data System is a resilient and secure repository for the data necessary to execute DNO applications. A fast storage system is an essential component of the HPCE: it is a local storage system for compute nodes, where applications as well as their current operational data (scratch data) are stored. Purpose of the Prototype The purpose of this prototype is two-fold. In the first place, it allowed both feasibility and implementation studies of the system architecture presented in [1], as it covers all major components of the HiPerDNO HPC Platform as defined in that document. In the second place, the prototype allows the testing of the DNO application scenarios to run on the full HPC system. Test applications of appropriate scheduling requirements and characteristics were employed during the demonstration (see chapter 4). It also allows the integration and test deployment of real DNO applications, which is the subject of ongoing work. Functional Description of HPC Prototype The HPC prototype consists of the HPCE and the DSS and it is depicted in Figure 1. One of the nodes in the HPCE is the head-node. In the prototype it plays also the role of the DMS, responsible for requesting execution of jobs in the HPC and it includes the scheduler and its required mechanisms. The HPC nodes require the presence of a common storage space (Fast Storage in figure 1). Figure 1. The HPC Prototype. FP

7 HPC Interface to the Data System All data are stored in the DSS. The DSS is not visible to the applications or indeed to either the DMS or the HPCE: data can only be accessed through appropriate client-server mechanisms. Both retrieval and storage of data are carried out through the client-server interface. In the prototype, data retrieval and storage results for the model applications are greatly simplified. We expect that in the HiperDNO HPC Platform appropriate data design, including metadata, will be carried out, but this is application-dependent and details will emerge in due course through the collaborations already started. The prototype relies on meta-data protocol that allows identification of the types of the application and of data access. Exchange of meta-data precedes exchange of content. The client-approach also allows changing the DSS technology (currently HDFS), without affecting the rest of the system and can be security hardened. Applications scheduling The scheduler ensures that applications are executed according to the execution priorities set by the DNOs. Based on collaboration with partners, we (WP1.2) have laid out the preliminary scheduler requirements for HPC applications, and are further discussed in chapter 4 below, and in [7]. We would like to stress that these requirements are not final: they corresponds to our current understanding, and no doubt will be modified following the DNOs feedback and the emergence of fully defined requirements as the project progresses. High-level requirements that must be supported by the scheduler are pre-emption and partitioning. Pre-emption allows the HPC system to stop a running task in order to make resources available for a high-priority task (kill-and-requeue). Low priority jobs are killed and re-added to the appropriate scheduler queue, without altering their job id, hence their place in the appropriate queue. Pre-empted tasks can then be restarted as soon as HPC Engine resources become available. Partitioning allows limiting the amount of resources that can be dedicated to jobs of a specific priority. The scheduler requirements are listed in greater detail in table 1. We have employed the Maui scheduler in the prototype, sitting on the Torque resource manager. Maui is a freely available, opensource package, and it supports all the requirements in table 1, with the exception of the optional no 6. Maui does not allow medium priority jobs to be both pre-emptable by higher priority jobs and at the same time pre-emptying lower priority ones: roles of pre-emptee and pre-emptor, so to speak, are incompatible, although this might be further studied. In any case, we do not believe that this has any major impact at this stage of the HiPerDNO project. The suggested partitioning of computing resources is adequate for current needs. Notice that we had to create a patch to ensure that Maui satisfies requirements no. 5 in table 1. This patch is of course part of this deliverable and is made available together will other sources, scripts and model applications. FP

8 No. Type Requirement 1. Resources Must The scheduler supports execution of jobs requiring 1...n compute nodes. 2. Pre-emption Must The scheduler supports pre-emption. 3. Pre-emption Must Jobs can be classified as critical and non-critical (critical jobs always pre-empt non-critical jobs). 4. Pre-emption Must Requeueing mechanism is supported, i.e. killing jobs and putting them at the top of their queue for execution. 5. Pre-emption Must Critical job can pre-empt as many low-priority jobs as necessary (but no more) to get the required resources for execution. 6. Pre-emption Nice to have Another class of jobs can be defined with intermediate priority level between critical and non-critical. These jobs can pre-empt non-critical jobs, however they can be preempted by critical jobs at the same time (pre-emption cascade). 7. Partitioning Must If no. 6 is not possible, the scheduler must allow the partitioning of resources. The amount of compute resources allocated to a certain class of critical jobs can be limited. 8. Scheduling latency Must To satisfy mission-critical requirements, high priority jobs must be run when they are submitted. Pre-emption and scheduling frequency must be sufficiently fine-grained to allow the stopping of low priority jobs and starting of high-priority jobs quickly and effectively. (Order of 1 second?) Table 1. HiPerDNO scheduler requirements. FP

9 3. Prototype Deployment Installation Tutorial The following sections discuss the installation of the HPC prototype. The prototype is to be deployed in a virtualized environment. We would like to stress that the target deployment infrastructure for the HPC system will be the real hardware. This is however the part of on-going work and not a part of this tutorial (see also Chapter 5). A number of prerequisites must be met before starting the HPC prototype installation. The deployment consists of two stages. The first stage involves the creation of a virtual cluster (nodes and network), the installation of the operating system and the network configuration. The second stage involves the installation and configuration of software which is central to the prototype: resource manager, scheduler, fast storage and data system software. Prerequisites The following list presents all necessary components for the deployment of the prototype. We suggest using VirtualBox as virtualization software and Ubuntu Server as an operating system for the nodes. Host o Hardware: Desktop PC (we used: Intel Core 2 quad, 8 GB RAM) o OS: Linux (we used: Ubuntu 10.04) Virtual Machines (VM) o Virtualisation software: Oracle VirtualBox ( o Guest OS: Ubuntu Server ( (or Ubuntu 10.04) Prototype software (Archive: D1.2.2.tar.gz from Basecamp), including o Cluster Resources MAUI v (patched version) o Cluster Resources Torque v o o o o o Creation of VM MAUI configuration file: maui.cfg Torque configuration file: torque.cfg demo scripts and applications a video of running prototype HDFS (Hadoop Yahoo distribution 20.2, hadoop.apache.org or Seven virtual machines should be created. The names listed below are fixed, as they are part of the prototype configuration: hpc1, hpc2, hpc3, hpc4, storage, hdfs1, hdfs2. The OS on the VMs do not require a GUI, and therefore have modest requirements: one CPU core and 384MB of RAM per virtual machine are sufficient. VirtualBox virtualization software provides graphical interface which makes it easy to create VM s quickly. During the creation most parameters can be left unchanged. During the VM creation the network interfaces must be set correctly. We chose to use two network interfaces per virtual machine. The first one gives access to the Internet (security updates, FP

10 software repositories). The second interface connects the VMs together using a virtual network and makes them accessible from the host. In terms of VirtualBox configuration, two network interfaces should be enabled during the VM generation: the first one will be a NAT (Network Address Translation)for Internet access, the second is a Host-Only-Adapter for the virtual network connection with other VMs and the host. The OS must then be installed. The installation of the Ubuntu Server is very straightforward, and it takes only few minutes. During installation, all the default options can be accepted. When asked, create the default username proto, and mark the OpenSSH package for installation. It is possible to speed up the whole process by cloning the virtual hard drive of the first VM and plugging it into the other six. It is important to remember to change their hostnames. For more information on cloning, please refer to the VirtualBox documentation [5, chapter 5]. Virtual network setup. This step assumes that all the VMs have been implemented, the OS has been installed in each of them and that the user proto is enabled and has rights to execute administrator s tasks (sudo). The network configuration requires editing two configuration files. The IP addresses are static for the virtual cluster network, and dynamic for the access to the Internet. The static addresses were chosen arbitrarily and they can be changed if necessary. The following section can be appended to each VM /etc/hosts file. // Prototype hostnames mapping hpc hpc hpc hpc hdfs hdfs storage The VirtualBox settings of each VM define the first network interface as NAT and the second as Host- Only-Adapter and the Linux kernel in the guest OS should assign to them eth0 and eth1 descriptors, respectively. The following section can be added to the /etc/network/interfaces file of every VM operating system if required. It may be necessary to substitute the IP address field for each VM to avoid any address conflicts. // The primary (NAT) network interface auto eth0 iface eth0 inet dhcp // The secondary (virtual network) interface auto eth1 iface eth1 inet static address netmask FP

11 After VMs are rebooted, the network should be up and running. In order to verify both interfaces, try to ping a machine on the internet and the host virtual network interface. It is suggested to enable SSH access on all nodes for the user proto, so that the nodes can be accessed from the host. By default the SSH service is enabled if the OpenSSH software was marked for installation during OS installation and we suggest to enable password-less SSH access from the host to all nodes. Instructions on how to do it can be found in the last section of this chapter, which discusses the installation of Data System software. This step completes the first stage of the installation. The Virtual cluster can be started and it is possible to connect to any VM machine from the host machine using SSH. In the next step, we are going to install the required software packages on the virtual nodes. Head-Node: hpc1 The hpc1 node is the head-node. It requires the following software packages to be installed: MAUI scheduler Torque server module Please use the MAUI installation package provided with this deliverable and not the one available on the website above: we have applied a required patch to the package we provided. MAUI Copy, unpack and install the file: maui-{version}.tar.gz, using the following commands: >tar xzvf maui-*.tar.gz >cd maui-* >./configure >make >make install Copy the file maui.cfg from the installation directory to the following location on the hpc1 node: maui.cfg -> (hpc1) /var/spool/maui/maui.cfg The MAUI installation and troubleshooting documentation can be found in [2]. Torque Copy, unpack and install the file: torque-{version}.tar.gz, using the following commands: >tar xzvf torque-*.tar.gz >cd torque-* >./configure >make >sudo make install The installation of Torque server module creates /var/spool/torque directory where configuration files are placed. From the installation folder torque files copy the following file: nodes -> (hpc1) /var/spool/torque/server_priv/nodes Set up Torque submission queues by copying and installing torque configuration file: torque.cfg -> (hpc1) ~/torque.cfg FP

12 >sudo qmgr < torque.cfg Torque installation and troubleshooting documentation can be found in [3]. Compute-Nodes: hpc1, hpc2, hpc3, hpc4 Torque The nodes hpc1,, hpc4 are compute nodes. Notice that hpc1 node is also the head-node. The compute nodes require the following software installed: Torque compute node module NFS client The Torque module will allow compute nodes to communicate with the head-node. To install the Torque compute node module login in the hpc1 node, enter the Torque installation director and create compute module installation packages: >cd torque-* >make packages It will generate a list of Torque packages. Copy and install the following packages on all four compute nodes: >sudo torque-package-mom-linux-i686.sh --install >sudo torque-package-clients-linux-i686.sh --install To enable Torque to run as a service, execute the following commands: >sudo cp contrib/init.d/debian.pbs_mom /etc/init.d/pbs_mom >update-rc.d pbs_mom defaults From the installation folder torque files copy the following file: server_name -> (hpc1,, hpc4) /var/spool/torque/server_name config -> (hpc1,, hpc4) /var/spool/torque/mom_priv/config NFS Client Compute nodes require the following software: NFS Client To install NFS Client on Ubuntu virtual machine, follow the following tutorial [6]. On each compute node: hpc1,, hpc4 create ~/shared/demo folder for the user proto. This folder should be mapped to the exported folder on the Fast Storage System (see next section). Fast Storage node: storage In the prototype dedicated node storage is used as a Fast Storage node and it requires: NFS Server Login to the storage node and install the NFS server following the tutorial [6]. FP

13 Create shared folder: >mkdir -p ~/storage/demo Export the folder as a shared folder, by adding the following entry to the /etc/fstab file: >/home/proto/storage/ /export/storage none bind 0 0 Verify whether exported folder can be accessed from all compute nodes. Data System nodes: hdfs1, hdfs2 The Data System requires installation of: HDFS (Hadoop Distributed File System) The HDFS package can be obtained from the Cloudera website under the following link: The HDFS has few prerequisites. It requires the JDK6 and SSH packages. The details of installing JDK6 are not addressed here as they are described in the link above. SSH setup requires password-less type of access between two machines. Setting up password-less access requires two steps: generating the key for the user, and copying this key to a remote machine for subsequent password-less access. To generate an encryption key, execute on both machines hdfs1 and hdfs2 the following command: >ssh-keygen t rsa When asked, confirm default parameters. To enable password-less SSH access between hdfs1 and hdfs2 machines, run the following commands on the nodes: On hdfs1 node: >ssh-copy-id I ~/.ssh/id_rsa.pub proto@hdfs2 On hdfs2 node: >ssh-copy-id I ~/.ssh/id_rsa.pub proto@hdfs1 Before the HDFS cluster can be used, it should be formatted. A list of useful HDFS commands, including the command to format the HDFS system is presented in the next chapter. FP

14 4. Testing the Prototype DNO-like model applications The prototype contains three model DNO applications as they are studied in the HiPerDNO project: distributed state estimation (DSE), condition-monitoring (CM), data-mining (DM). These applications are different with respect to criticality, execution pattern (periodic, aperiodic) and required compute resources. The proposed characteristics are summarized in table 2 were chosen as an example scenario. At the same time they are based on the input provided by the partners in the project. DNO Application DSE CM DM Scheduling characteristics mission-critical, periodic, short submission interval, parallel High-priority, periodic, long submission interval, batch of serial tasks non-critical w.r.t timely results, aperiodic, long-execution time Table 2. Model DNO application, execution requirements. The DSE is a periodic job with a short submission interval (few minutes; for the purpose of this prototype shortened significantly). It is a parallel task requiring a number of compute nodes and it is a mission-critical task, thus requiring immediate execution at the moment of submission. The CM is a periodic task with long submission intervals. The CM submission consists of a batch of small serial jobs submitted together at regular intervals. CM jobs are high priority and are executed on a limited number of compute nodes The DM jobs are not periodic; they are serial tasks, submitted at arbitrary intervals, requiring long execution times. They are not critical jobs from the DNO point of view and can be executed when the system load due to higher priority jobs allows. Thus their execution can be postponed. The model DM applications includes checkpointing/warm restarting, thus allowing DM jobs to pre-empted and rescheduled (kill-and-requeue). Running the example In the previous chapter HPC hardware infrastructure and software components (Torque, MAUI, NFS, HDFS) were set up. Before the prototype can be run the DNO model applications and submission scripts must be deployed from the host, as well as basic data layout in the DSS. Start the prototype by executing the following script on the host: >./startvms When all virtual machines are running, deploy all scripts and DNO model applications from the host by running: >./deploy The prototype can be shut down (later) by running the following command from the host: >./stopvms FP

15 Open the terminal window and log into the hdfs1 node. Create data folders for DNO applications: >hadoop dfs mkdir DSEIn >hadoop dfs mkdir DSEOut >hadoop dfs mkdir CMIn >hadoop dfs mkdir CMOut >hadoop dfs mkdir DMIn >hadoop dfs mkdir DMOut Next, generate example input files for DNO applications by running: >./set.py DSE 10 >./set.py CM 2 >./set.py DM 500 These commands will create the input files needed by applications running in the prototype. The second parameter behind the name of the application (DSE, CM, DM) specifies the length of its input file and effectively the execution time of the application. Run the Data Store server script: >./ds.py The script is a part of the client-server interface, which sits on top of the Data System. Open the terminal window and log into the node hpc1. Run the following script: >./monitor.sh Open the second terminal window and log into the node storage. Run the following script: >./filemon.sh The two terminal windows will show scheduling and checkpoint information accordingly. Open another terminal window and login into hpc1 node. Run the prototype demonstration with the following command: >./demo.sh The demo takes a couple of minutes to execute. In a terminal window with scheduling information, one can see which jobs are submitted but waiting for execution, which are executing and on which nodes. The second terminal window shows how checkpoint files are created. Both terminal windows are shown in figure 2, which presents a snapshot from the running prototype. We can see that five jobs are present in the HPC system: four DM jobs and one DSE job. Three DM jobs are not running because of the DSE job pre-emption. Only one DM job is in execution. In the right terminal window we can see that new checkpoint files are being generated for the DM job running. FP

16 Figure 2. HPC Prototype demo. Other key commands (HDFS, Torque, MAUI) In order to manage the prototype the following commands may prove to be useful. More complete list of commands to manage HDFS, Torque, MAUI can be found through the references [2-4]. HDFS The basic operations on HDFs include formatting the HDFS file system, copying, listing, and removing the files. To format the HDFS, using the following command: >sudo hadoop namenode format To display the list of files in the HDFS directory issue the following command: >hadoop dfs ls URI To remove a file from the HDFS, issue the following command: >hadoop dfs rm URI To display the content of the file onto a terminal, issue the following command: >hadoop dfs cat URI To copy a file from the local file system to HDFS, use the following command: >haddop dfs copyfromlocal {localsrc} URI To copy a file from HDFS to a local file system, use the following command: >hadoop dfs copytolocal URI <localdst> For the complete list of HDFS commands please refer to [4]. Torque By default, Torque starts its standard PBS scheduler when the head node boots up. The PBS scheduler is not part of the prototype and it should be stopped manually, before enabling MAUI (see the next section). To stop the PBS scheduler, type the following command on the hpc1 front node: >sudo pkill pbs_sched How to start MAUI scheduler is described further on. In order to verify whether the HPC compute nodes FP

17 are running, use the following command on the hpc1 front node: >pbsnodes In the output of this command check if for each node the state is set to free. It if is, it means the node is up and running and waiting for jobs to be submitted. In order to check the list of submitted jobs use the following command on the head node HPC1: >qstat The command will display the list of jobs submitted to the resource manager, together with their state. MAUI The key configuration tool for the MAUI scheduler is its configuration file. It is not the purpose of this document to go in depth on the configuration parameters of the MAUI schedulers. For those interested, refer to [2]. The configuration file maui.cfg is provided together with this deliverable and it should be copied to appropriate location on the HPC1 head node before starting MAUI scheduler. The location for MAUI configuration file is: /var/spool/maui/maui.cfg. Make sure that the default PBS scheduler is not running (see previous section). To start MAUI use the following command on the HPC1 head node: >maui Editing schedules The prototype is shipped with an example scenario demo.sh. It is possible to submit CM, DM, DSE jobs without demo.sh script. In order to submit CM or DM type: >./submit.sh DM >./submit.sh CM To submit DSE, one can specify additionally how many nodes DSE job would require (max 3). >./submit.sh DSE 2 >./submit.sh DSE 3 To influence the runtime of the jobs modify the size of the input file stored in the Data Store (see section Data System setup in the previous chapter). For example to shorten by half the execution time of the DSE execute the following command on the hdfs1: >./set.py DSE 5 FP

18 5. From Prototype to System Deployment The HPC Prototype described in this document lays the ground for the work ahead in the WP1.2 and other work packages in the HiPerDNO project. We are planning to deploy the current HPC prototype on real hardware. This deployment will be made available to partners in the HiPerDNO project. Based on initial discussions within the OeRC team we will make available an Intel-based cluster consisting of seven or eight dual-cpu nodes. It will be configured so as to allow both the development and tuning of real applications as well as study of the overall HPC System under realistic deployment conditions. We plan to use Rocks ( to ease the deployment and maintenance of the system, and interconnects adequate to the task in hand (hopefully, 10 Gbit Ethernet). Work on the deployment will commence in August and we plan to make it available to all partners in September. We are also starting collaborations with various HiPerDNO partners (EDF, BU, Indra/UF) with the aim of porting to and integrating their algorithms on the HPC platform. More information can be obtained from the forthcoming HiPerDNO deliverable ([7]). FP

19 6. References 1. S. Salvini, P. Lopatka, D. Wallom, HiPerDNO Deliverable D Report on the architecture and performance criteria used for selection. 2. Adaptive Computing Enterprises, Maui Scheduler Administrator s Guide version 3.2, 3. Cluster Resources, TORQUE Admin Manual version 3.0, 4. The Apache Software Foundation, HDFS User Guide., 5. Oracle Corporation, Oracle VM VirtualBox, 6. Ubuntu documentation, Setting up NFS HowTo, 7. HiPerDNO project deliverable, D3.1.1, Detailed specification report on HPC architecture and platform standardisation to support the development of novel DMS functionality FP