Using the Windows Cluster

Using the Windows Cluster Christian Terboven terboven@rz.rwth aachen.de Center for Computing and Communication RWTH Aachen University Windows HPC 2008 (II) September 17, RWTH Aachen

Agenda o Windows Cluster Overview o Login to the Cluster o File Storage Strategies o Using the Batch System cluster win: Windows Compute Cluster Server cluster win beta: Windows HPC Server 2008 2

HPC on Windows: Objectives o Windows HPC is a regular service of the Computing Center: Account provisioning via TIM one account (login+pw) for Solaris, Linux and Windows Files on Unix are accessible from Windows because of same file infrastructure ($HOME = H:, $WORK = W:) Operating Model: Interactive Machines + Batch System Programming and Software Support: Languages: C, C++, Fortran (, Java, C#) Parallelization: MPI, OpenMP, Intel TBB, Threading ISV Codes: Matlab, Ansys, numerical libraries, User training on all platforms! 3 o HPC service is open for employees and students as well!

Current Hardware of the Windows Cluster o 28 Fujitsu Siemens Primergy RS 200 S4 servers 2x Intel Xeon E5450 (quad core, 3.0 GHz) 16 GB main memory per server 4x DDR InfiniBand network Latency: < 5 us Bandwidth: 1200 MB/s In total over 1.5 TFLOP/s peak performance o Number of machines on Solaris, Linux and Windows is adjusted to the user demand At least 16 machines running Windows 4 o All machines run 64bit Windows!

Windows Cluster: Frontends o Currently we are running three Frontends for two Clusters: cluster win beta. rz.rwth aachen.de Windows Server 2008 Beta (ISV) Software cluster win. rz.rwth aachen.de Windows Server 2008 Stable (ISV) Software Load Balancing cluster win2003. rz.rwth aachen.de Windows Server 2003 For: ANSYS + cygwin Microsoft Windows HPC Server 2008 (2008er Cluster) HN: WINWCC01 Microsoft Compute Cluster Server (2003er Cluster) HN: WINHTC01 5 Batch Job Submission

Cluster Map of 09/2008 (for reference) 6

Login from Windows (1/3) o Use the Remote Desktop Connection program, usually available under All Programs Accessoires Communication. Specify the computer name, in general cluster-win.rz.rwthaachen.de. 8 You can save a set of settings under a named profile / link.

Login from Windows (2/3) You can choose a resolution or fullscreen mode. You can choose the color depth. In fullscreen mode you should set this flag to ease the handling of the remote desktop window. 9

Login from Windows (3/3) Take resources of your local computer with you: Audio device Printer Clipboard Local hard disc drives Locally mounted network drives 10

Login from Linux o Use the rdesktop program available from www.rdesktop.org, probably already included in your distribution. 11 o Basic usage: rdesktop [options] host with u <user> Login as user <user> d WIN HPC Login to domain WIN HPC 4 Use protocol version 4 (often needed) g WxH Use resolution Width x Height f Use fullscreen resolution a 24 Use 24bit color depth k de Use german keyboard layout r sound:local Play sound on local system

Login to Windows Server 2008 New Remote Desktop Connection program (e.g. with Vista) allows the specification of username before login. 12 If no domain selection is possible Username: WIN HPC\... Or: Specify in rdesktop program

File Storage Strategies (1/2) o Batch jobs cannot access H: (= $HOME on Unix) or W: (= $WORK on Unix) directly via the drive letter: Network paths have been unified H: is \\cifs\cluster\home\userid W: is \\cifs\cluster\work\userid C:\Shared_Software is \\cifs\cluster\software Either use those, or connect manually in batch script, e.g. net use H: \\cifs\cluster\home\userid 14 o On H:\ and W:\ in hidden subdirectory.zfs you will find regular snapshots Old version of that directory/share, last three days, at 12:00h and 24:00. Can be used instead of restore request from backup service.

o File Storage Strategies (2/2) D:\Temp\userid (=%TMP%) might vanish between logins! o Windows only file storage: Documents Windows native file server for faster metadata operations Accessible via Explorer / File Dialogs: User Name \Documents Network drive X:\ \\cifs\cluster\documents\ct747764\ Not accessible from Unix! 15 o All directories/shares have a quota, that is a size limitation. Need more space? email hpc@rz.rwth aachen.de

Using the Batch System (1/6) o Find the Compute Cluster Job Manager under All Programs Microsoft Compute Cluster Pack Compute Cluster Job Manager. Only possible to influence your own jobs! 17 Show Filter: All Jobs Active Jobs Finished Jobs Failed Jobs Cancelled Jobs

Using the Batch System (2/6) o To submit a new Job choose File Submit Job. You are free to choose a Job Name and a Project Name as you like. 18

Using the Batch System (3/6) o Several options can be set for a Job. You have to specify the number of processors your job needs. You may estimate the runtime foryourjob, but you are not required to do so. Currently, there is no runtime limit for a Job / Task. 19 Problem without runtime limit: Malfunctioning jobs may be overlooked. We cannot save Jobs from dying in case of system maintenance.

20 High Performance Computing on Windows Using the Batch System (4/6) o A Job consists of one or more Tasks. Tasks aretheexecutable pieces of a job. You can freely choose a Task Name. For Command Line you can specify the full path to a program including program options or to a.bat or.cmd file. You have to use network paths instead of drive letters (\\cifs\cluster\home\... instead of H:) in any path. For MPI Tasks just include mpiexec in the Command Line, do not specify any other MPI options.

21 High Performance Computing on Windows Using the Batch System (5/6) o Several options can be set for a Task. Tasks should not use any GUI elements. You can specify Standard Input, Standard Output and Standard Error for a Task, as well as the Working Directory. Using Task Dependencies you can define an order in which Tasks have to be executed. In the Environment register, you can set environment variables for a Task, e.g. OMP_NUM_THREADS for OpenMP.

Using the Batch System (6/6) o A dedicated set of machines can be specified for a Job / Task. If you do not need a special machine, just leave Use any available nodes option set. Problem for Shared Memory parallel Jobs: if you specify n processors, the assigned n processors may be distributed among several machines. 22

Using the New Batch System (1/5) o Find the HPC Job Manager under All Programs Microsoft HPC Pack HPC Job Manager. Several pre conf. views available. Modern Action Pane. 24

25 High Performance Computing on Windows Using the New Batch System (2/5) o To submit a new Job choose Actions Job Sumission New Job. You are free to choose a Job Name and a Project Name as you like. You might specify runtime and failure options for the job. Resource allocation changed significantly: Per Core, or Per Socket, or Per Node. Saving of Job Templates possible.

26 High Performance Computing on Windows Using the New Batch System (3/5) o Resource Allocation Granularity: Per Core: Get n processor cores. No further restrictions, for example it cannot be assumed that a (sub)set of cores shares the same main memory ( not suited for Shared Memory). Per Socket: Get n sockets. On our cluster, currently each socket has four cores (quad core Xeon), thus it can be used for Hybrid or Shared Memory (up to four threads per process). Per Node: Get n nodes. On our cluster, currently each node has two sockets à four cores (dual socket quad core Xeon), thus it can be used for Hybrid or Shared Memory (up to eight threads per node). o If you use OpenMP: Set OMP_NUM_THREADS env. variable, otherwise you will get #cores threads mpiexec genv OMP_NUM_THREADS 2

27 High Performance Computing on Windows Using the New Batch System (4/5) o A Job consists of one or more Tasks. Command Line: You can specify the full path to a program including program options or to a.bat or.cmd file. You have to use network paths instead of drive letters (\\cifs\cluster\home \... instead of H:) in any path. For MPI Tasks just include mpiexec in the Command Line, do not specify any other MPI options.

Using the New Batch System (5/5) o Some restrictions for the node selection can be specified. Allow selected classes of nodes only. Allow a selected set of nodes only. 28 Allow nodes with enough memory only.

Demo: Using the (New) Batch System Windows Compute Cluster Server (2003) via cluster-win.rz.rwth-aachen.de and Headnode WINHTC01 Windows HPC Server 2008 via cluster-win-beta.rz.rwth-aachen.de and Headnode WINWCC01 29

The End Thank you for your attention! Questions? 30