Using Huygens. Jeroen Engelberts - jeroene@sara.nl Consultant Supercomputing. UVA HPC Cursus Using Huygens

Transcription

1 Using Huygens Jeroen Engelberts - jeroene@sara.nl Consultant Supercomputing

2 Overview Course SARA HPC facilities Huygens System description Endianness File systems Module environment Batch system Accounting User Support Hands on Let's Play!

3 About SARA SARA is an independent non-profit company delivering a full service package in the areas of High-Performance Computing and Networking, data storage, visualization and e-science services Founded in 1971 as an Amsterdam computing center by the two Amsterdam universities (UvA and VU) and the current CWI Independent as of 1995 Founded Vancis in 2008 offering ICT services and ICT products to enterprises, universities, and educational and healthcare institutions. First supercomputer in The Netherlands at SARA in 1984 (Control Data Cyber 205). Hosting the national supercomputer(s) ever since.

4 SARA OSD Formerly known as HPC&V and HPN, Operations, Support and Development is subdivided in six groups: Supercomputing Clustercomputing e-science & Cloud Services Visualization Data services Network innovation & support Some 50 people System Programmers / Consultants BSc - MSc - PhD

5 HPC Facilities at SARA Capacity computing BiG Grid national grid infrastructure Lisa national compute cluster Capability computing Huygens national supercomputer Experimental capability computing GPU cluster HPC Cloud Access method depends on your affiliation, for questions contact: SARA helpdesk

6 Supercomputing Huygens IBM pseries 575 Processor: IBM Power6, 4.7 GHz (dual core) Node: 16 dual core processors In total: 108 nodes, 3456 cores 65 Tflop/s Memory: 128 GB (90 SCN) or 256 GB (18 MCNs) TB Interconnect: 8 (4 DDR) Infiniband 160 Gbit/s inter-node Disk: 3240 x 300 GB = 972 TB (700 TB net capacity) OS: SuSE SLES11SP1 Watercooling: 33% more energy efficient (6º C inlet; 25º C outlet) Energy consumption: kw Cabling: 30.7 km (IB, FC, UTP) Huygens 9 compute racks (1700 kg each) 4 Infiniband racks 20 storage racks Number 52 in the November 2008 Top500 (48.55 Tflop/s) 79 in Jun09, 92 in Nov09, 109 in Jun10, 156 in Nov10, 303 in Jun11, Designed to be a well balanced system instead of being a Top500 killer!

7 Huygens Water Cooling

8 Internode Networking Huygens equipped with Infiniband network, each node has 2 Infiniband adapters. Each adapter has 4 ports. The 8 ports cabled to 8 separate switches. Port has throughput of 20 Gbit/s, aggregated to 160 Gbit/s. This comes down to 2 GB/s, aggregated to 16 GB/s.

9 Processors and Memory (1) NUMA (Non uniform memory architecture) effects occur in Huygens due to design. Each node consists of four quads, or multi-chip module (MCM). Each quad has four processors. Each processor is a dual chip module (DCM). Three levels of cache memory L1 (64kB instruction, 64kB data) on core L2 (4 MB) on chip L3 (32 MB) on MCM

10 Processors and Memory (2) POWER6 chip 32 MB L3 8B x2 rd 8B x2 wr 4B/8B x2 Core Ghz Core Ghz 128kB L1 128kb L1 4 MB Private L2 4 MB Private L2 L3 Control Directory I/O Ctlr 4B read 4B write Fabric Switch Memory Ctlr I/O Interface 4B/8B x2 Off-Node Fabric 8B x2 read 4B x2 write 2B/8B x2 2B/8B x2 2B/8B x2 On-Node Fabric Memory

11 Affinity It is possible to assign tasks to specific cores manually. On Huygens, affinitization is automatically done by LoadLeveler submit filter from SARA. More details on affinity later today.

12 Endianness (1) Origin: Gulliver s Travels, dispute between Lilliput (GB) and Blefuscu (F) regarding the opening of eggs introduced in ICT by Danny Cohen in 1980 How the hexadecimal value 0x89ABCDEF is stored:

13 Endianness (2) Huygens Big Endian PCs (Intel, AMD) Little Endian Transport of binary data files not trivial per se Options Text files instead of binary data files. NetCDF (Network Common Data Form) HDF (Hierarchical Data Format) XLFRTEOPTS=ufmt_littleendian=unit list (runtime)

14 File systems cabinets

15 File systems /home home directory of user Quotum (default) 200 GB / user Being backed up Divided in 9 parts of 14 TB / each Aggregated speed 540 MB/s One such part can be restored in a day from tape /scratch scratch directories of user Quotum 8 TB on /scratch/local and 8 TB on /scratch/shared Not being backed up /scratch/shared available to all nodes /scratch/local available to each node separately /scratch/shared aggregated speed of 26.4 GB/s /scratch/local aggregated speed of 21 GB/s Retention time: at least two weeks (currently)

16 File systems /project project space Special to be requested for projects Not being backed up Divided in 2 parts, one slow one fast Capacity/Aggregated speed Slow: 132 TB 5.3 GB/s Fast: 211 TB 21 GB/s /archive Directory for archive purposes Disk cache with tape robot Older files take long to retrieve Virtually no limit tapes can be added Preferably store large files tar, zip

17 Modules Why modules? Environment variables are set for you, like: PATH LD_LIBRARY_PATH Multiple versions of software can coincide

18 Modules Commands module avail module load modulename module add modulename module display modulename module unload modulename module rm modulename module list module help

19 Module avail

20 Module load

21 Module display

22 Module unload / Module list

23 LoadLeveler Batch environment Jobs Commands and Keywords Scheduling policy Submitfilter

24 LoadLeveler Batch environment Preparations interactive Computations batch environment Description of work in a job script submit On Huygens LoadLeveler. See: Tivoli Workload Scheduler Loadleveler (Using and administering). LL controls: resource allocation scheduling of jobs execution of jobs keeping track of job information. Huygens configuration for batch: 1 Interactive node 2 testnodes 101 batch nodes (18 with 256 GB of memory all others with 128 GB of memory).

25 LoadLeveler Job script Job script: Number of LoadLeveler keywords (starting with ) describing resource requirements to set up optimal environment for your job. Closed by queue Shell script that describes the setup of the computation. Creating a working directory and go to this directory Copy input files and executable to working directory Invoke your executable Copy output files to home directory or archive. A number of examples can be found at: Copy-paste them and adapt them to your situation.

26 LoadLeveler Job example #!/bin/bash node = 2 tasks_per_node = 32 notification = never input = /dev/null output = out.$(jobid) error = err.$(jobid) wall_clock_limit = 1:10:20 job_type = parallel network.mpi = sn_all,not_shared,us queue # Here the shell script starts. cd $HOME/workdir./my_program

27 LoadLeveler Commands Intrinsic LoadLeveler commands llsubmit: submit a job llcancel: cancel a job llq: query LoadLeveler on job status llstatus: provide status of nodes Additional SARA scripts (to be found in /usr/sara/bin), providing extra views: llstx: give node occupation llstc: give class configuration of Huygens lljobs: alternative for llq

28 LoadLeveler Batch classes A Job is queued in a class by LoadLeveler (not in job script), based on user specified node usage, wall clock limit and number of nodes #@ node_usage = shared/not_shared #@ wallclock_limit = hh:mm:ss #@ nodes = 4 To allow LoadLeveler scheduler maximum freedom only a few classes are defined: (shared, short, normal, special). Use llstc to view the current configuration in classes:

29 LoadLeveler Batch classes = unknown 1 = normal 2 = shared 3 = short 4 = spec_sys 5 = spec_training

30 LoadLeveler Scheduling policy Users can submit an unlimited number of jobs at one time. When a job is submitted it gets status: Queued (Eligible for scheduling) Not Queued (Not eligible for scheduling). Jobs are executed according to scheduling policy: First in - First out BUT: Only jobs in queued status are considered Timestamp when job acquires queued status is considered to be submission time. Number of jobs in queued status is limited Backfill mechanism applies

31 LoadLeveler Scheduling policy Configurable limits (by LoadLeveler administrator): Maxqueued: maximum number of jobs per user that can have queued status (default 8). Maxrunning: maximum number of jobs per user that can be running simultaneously (default 8). Maximum wallclock limit: (currently 5 days). Maximum number of nodes: (currently 80 in normal production mode).

32 LoadLeveler Submitfilter When you submit a job it is parsed by a submit filter (Huygens specific). It invokes the budget check script: Checks the remaining budget Computes the number of PNU s Of all jobs that have finished the same day, All jobs that are still in the queue It determines the class from the wallclock limit, number of nodes and node_usage. It checks whether or not some keywords are set. If some keywords are not set it sets them to a (sensible) default value. Defaults can be overruled by #@ skip_rset

33 Accounting Getting access to Huygens Accounts and Logins Budget and jobcost

34 Accounting Getting access to Huygens Take a look at the SARA website: Proposal to NWO Filling in the forms in IRIS Peer review process Approval from NWO, What next? Granting letter (from NWO) A copy to SARA Acceptance letter (from NWO) Fill it in and return it to NWO. User form (download from SARA website) Fill in and send it to SARA. Usage agreement (download from SARA website) each user should fill this in, sign it and send it to SARA.

35 Accounting Accounts and Logins After receiving the proper forms SARA will create an account and login for you. Account: administrative entity to keep track of used budget Owner of an account is the PI (Principal Investigator) who submitted the project proposal to NWO. A project can have one or more accounts associated with it Each account can have several logins coupled to it. Duration of an account is 1 year (expiration date set by NWO) Login: combination of username + password and environment to give physical access to Huygens. Logins are STRICTLY PERSONAL. A login is at any time associated with one and only one account Logins can be moved from one account to another

36 Accounting Account expiration Monthly warnings will be sent (to PI!!!) that account will expire, starting 3 months before expiration date. Extension of an account is possible (contact NWO). Asking for extra time (budget will remain) Asking for extra budget (will be added to remaining budget) Submit a continuation proposal for extension of the same project (budget will be reset to new value). Submit a completely new proposal with new accounts (logins can be moved to new account). After expiration date the Account will be blocked Login to Huygens is denied You will be asked to give SARA permission to remove the login and all data associated with it from the system If you don t respond we will first seek permission of the Account Owner to remove everything. If still no response we will remove everything after a grace period of 6 months (in Usage Agreement).

37 Accounting Budget and jobcost Budget: If project proposal is accepted a budget is assigned to the accounts. Budget is expressed in PNU (Processor Node Uur) or equivalently in SBU (System Billing Units). 1 PNU = the use of 1 core for 1 hour on Huygens. Jobcost (Shared nodes) Jobcost based on cpu time Jobcost (Dedicated nodes) Jobcost based on wallclock time: You always pay for a complete node. Using 1 node for 1 hour will cost you 32 PNU (regardless of how many processes are running on the node). Capacity of Huygens: Gross: ~ 28M PNU/year. Nett: ~ 24M PNU/year (2M/month)

38 Accounting Budget and jobcost For overview of jobcosts use the command accuse Gives consumed budget per day or per month. For overview of budget use the command accinfo Information about initial, consumed and remaining budget. Gives contact information ( address of account owner) Gives list of logins associated with the account.

39 Accounting Budget and jobcost Account : sscpwimr (HUYGENS2) Customer : (10305) Klant voor subinstelling wimr@sara.nl Initial budget ; 60944:13 Used budget ; 17659:03 Remaining budget ; 43285:10 Creation date ; Last modified ; Valid until ; User ID(s) linked to this account: User Group wimr ANY wimr2 ANY

40 Accounting Budget and jobcost Accounting information Jobinfo is kept by Loadleveler in a temporary file After the job finishes: Correct finish: the temporary file is added to a history file. Loadleveler crash: the temporary file is discarded Job restarted by system: temporary file is discarded Once a day (during the night): accounting information is extracted from this history file and added to the accounting database. The remaining budget is computed: If this is negative your account will be blocked. Budget check: To avoid that you will overtax your budget we introduce the budgetcheck, that will run at submission time and at job start. When remaining budget is not sufficient, your job will be refused.

41 User Support Expert user support (help desk, online documentation, (grid-) tutorials, courses) 7 full time HPC consultants (all computational scientists) for both Huygens and Lisa Optimization, parallelization/hyperscaling of user codes Benchmarking Visualization and data analysis of complex data sets Grid support Gridification of applications Grid-enabling of (local) resources Grid tutorials, education of traditional HPC users to use grid infrastructure Support for Grand-Challenge applications e-science support

42 Hands-On Contents Download necessary files locally Install user tools (Windows users only) Copy files to Huygens Login to Huygens Compile Molden (a comp. Chemistry tool) Look at input file with Molden Submit a job (geometry optimization) wait for the result... Analyze the result Copy back output/results locally

43 Hands-On Download Download the material from ftp://ftp.sara.nl/pub/outgoing/usinghuygens It includes: molden5.0.tar.gz Molecular Visualization Tool molecule.job, molecule.zmat Input for example For Windows users additionally: putty-0.62-installer.exe winscp437setup.exe Xming setup.exe For Windows users: Install the three packages (mentioned above)

44 Hands-On Copy files to Huygens Mac & Linux users: Open a Terminal (Linux) or X11 (Mac) Go to the directory where you downloaded files Type: scp molecule.* molden5.0.tar.gz sdemonnn@huygens.sara.nl: where nnn is your demo number For Windows users: Start WinSCP Create New and fill in: Host name: huygens.sara.nl User name: sdemonnn Password: ******* Look up downloaded files and copy them to Huygens

45 Hands-On Login to Huygens Mac & Linux users: Open a Terminal (Linux) or X11 (Mac) Type: ssh -X sdemonnn@huygens.sara.nl where nnn is your demo number For Windows users: Start Xming (if not yet started system tray) Start PuTTY Host Name: huygens.sara.nl Click on Connection/SSH/X11 Check under X11 forwarding Enable X11 forwarding Click Open User your sdemonnn username and password to login

46 Hands-On Molden All users: Extract Molden tarball: tar zxf molden5.0.tar.gz Go into Molden directory: cd molden5.0 Modify makefile: gedit makefile change CC = cc into CC = gcc Make the binary: make Move the resulting binary to ~/bin: mv molden../bin Go back to home directory: cd.. Have a look at the molecule: molden molecule.zmat

47 Hands-On Inspect job (1) All users: Edit job script: gedit molecule.job

48 Hands-On Inspect job (2) LoadLeveler instructions: #!/bin/bash node = 1 tasks_per_node = 32 input = /dev/null output = molecule.$(jobid).out error = molecule.$(jobid).err wall_clock_limit = 1:00:00 # notification = never # job_type = parallel network.mpi = sn_all,not_shared,us # ll_res_id = p r queue

49 Hands-On Inspect job (3) Shell commands: STARTDIR=`pwd` echo "%NProcShared = 32" > $TMPDIR/molecule.inp echo "#RHF/3-21G Opt" >> $TMPDIR/molecule.inp echo "" >> $TMPDIR/molecule.inp echo "My molecule" >> $TMPDIR/molecule.inp echo "" >> $TMPDIR/molecule.inp echo "0,1" >> $TMPDIR/molecule.inp cat molecule.zmat >> $TMPDIR/molecule.inp cd $TMPDIR module load gaussian/09 g09 < molecule.inp

50 Hands-On Submit job All users: Submit job: llsubmit molecule.job Inspect status of your job: llq -u sdemonnn Once running, inspect ouputfile: tail -f molecule.<jobid>.out fill in job_id Once finished, analyze outputfile: molden molecule.<jobid>.out In Molden, press Movie See how benzene becomes flat and hexagonal!

51 Thank you for listening!