Using Huygens Jeroen Engelberts - jeroene@sara.nl Consultant Supercomputing
Overview Course SARA HPC facilities Huygens System description Endianness File systems Module environment Batch system Accounting User Support Hands on Let's Play!
About SARA SARA is an independent non-profit company delivering a full service package in the areas of High-Performance Computing and Networking, data storage, visualization and e-science services Founded in 1971 as an Amsterdam computing center by the two Amsterdam universities (UvA and VU) and the current CWI Independent as of 1995 Founded Vancis in 2008 offering ICT services and ICT products to enterprises, universities, and educational and healthcare institutions. First supercomputer in The Netherlands at SARA in 1984 (Control Data Cyber 205). Hosting the national supercomputer(s) ever since.
SARA OSD Formerly known as HPC&V and HPN, Operations, Support and Development is subdivided in six groups: Supercomputing Clustercomputing e-science & Cloud Services Visualization Data services Network innovation & support Some 50 people System Programmers / Consultants BSc - MSc - PhD
HPC Facilities at SARA Capacity computing BiG Grid national grid infrastructure Lisa national compute cluster Capability computing Huygens national supercomputer Experimental capability computing GPU cluster HPC Cloud Access method depends on your affiliation, for questions contact: SARA helpdesk hic@sara.nl
Supercomputing Huygens IBM pseries 575 Processor: IBM Power6, 4.7 GHz (dual core) Node: 16 dual core processors In total: 108 nodes, 3456 cores 65 Tflop/s Memory: 128 GB (90 SCN) or 256 GB (18 MCNs) 15.75 TB Interconnect: 8 (4 DDR) Infiniband 160 Gbit/s inter-node Disk: 3240 x 300 GB = 972 TB (700 TB net capacity) OS: SuSE SLES11SP1 Watercooling: 33% more energy efficient (6º C inlet; 25º C outlet) Energy consumption: 570 600 kw Cabling: 30.7 km (IB, FC, UTP) Huygens 9 compute racks (1700 kg each) 4 Infiniband racks 20 storage racks Number 52 in the November 2008 Top500 (48.55 Tflop/s) 79 in Jun09, 92 in Nov09, 109 in Jun10, 156 in Nov10, 303 in Jun11, Designed to be a well balanced system instead of being a Top500 killer!
Huygens Water Cooling
Internode Networking Huygens equipped with Infiniband network, each node has 2 Infiniband adapters. Each adapter has 4 ports. The 8 ports cabled to 8 separate switches. Port has throughput of 20 Gbit/s, aggregated to 160 Gbit/s. This comes down to 2 GB/s, aggregated to 16 GB/s.
Processors and Memory (1) NUMA (Non uniform memory architecture) effects occur in Huygens due to design. Each node consists of four quads, or multi-chip module (MCM). Each quad has four processors. Each processor is a dual chip module (DCM). Three levels of cache memory L1 (64kB instruction, 64kB data) on core L2 (4 MB) on chip L3 (32 MB) on MCM
Processors and Memory (2) POWER6 chip 32 MB L3 8B x2 rd 8B x2 wr 4B/8B x2 Core 1 4.7 Ghz Core 2 4.7 Ghz 128kB L1 128kb L1 4 MB Private L2 4 MB Private L2 L3 Control Directory I/O Ctlr 4B read 4B write Fabric Switch Memory Ctlr I/O Interface 4B/8B x2 Off-Node Fabric 8B x2 read 4B x2 write 2B/8B x2 2B/8B x2 2B/8B x2 On-Node Fabric Memory
Affinity It is possible to assign tasks to specific cores manually. On Huygens, affinitization is automatically done by LoadLeveler submit filter from SARA. More details on affinity later today.
Endianness (1) Origin: Gulliver s Travels, dispute between Lilliput (GB) and Blefuscu (F) regarding the opening of eggs introduced in ICT by Danny Cohen in 1980 How the hexadecimal value 0x89ABCDEF is stored:
Endianness (2) Huygens Big Endian PCs (Intel, AMD) Little Endian Transport of binary data files not trivial per se Options Text files instead of binary data files. NetCDF (Network Common Data Form) HDF (Hierarchical Data Format) XLFRTEOPTS=ufmt_littleendian=unit list (runtime)
File systems cabinets
File systems /home home directory of user Quotum (default) 200 GB / user Being backed up Divided in 9 parts of 14 TB / each Aggregated speed 540 MB/s One such part can be restored in a day from tape /scratch scratch directories of user Quotum 8 TB on /scratch/local and 8 TB on /scratch/shared Not being backed up /scratch/shared available to all nodes /scratch/local available to each node separately /scratch/shared aggregated speed of 26.4 GB/s /scratch/local aggregated speed of 21 GB/s Retention time: at least two weeks (currently)
File systems /project project space Special to be requested for projects Not being backed up Divided in 2 parts, one slow one fast Capacity/Aggregated speed Slow: 132 TB 5.3 GB/s Fast: 211 TB 21 GB/s /archive Directory for archive purposes Disk cache with tape robot Older files take long to retrieve Virtually no limit tapes can be added Preferably store large files tar, zip
Modules Why modules? Environment variables are set for you, like: PATH LD_LIBRARY_PATH Multiple versions of software can coincide
Modules Commands module avail module load modulename module add modulename module display modulename module unload modulename module rm modulename module list module help
Module avail
Module load
Module display
Module unload / Module list
LoadLeveler Batch environment Jobs Commands and Keywords Scheduling policy Submitfilter
LoadLeveler Batch environment Preparations interactive Computations batch environment Description of work in a job script submit On Huygens LoadLeveler. See: http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Tivoli Workload Scheduler Loadleveler (Using and administering). LL controls: resource allocation scheduling of jobs execution of jobs keeping track of job information. Huygens configuration for batch: 1 Interactive node 2 testnodes 101 batch nodes (18 with 256 GB of memory all others with 128 GB of memory).
LoadLeveler Job script Job script: Number of LoadLeveler keywords (starting with # @ ) describing resource requirements to set up optimal environment for your job. Closed by # @ queue Shell script that describes the setup of the computation. Creating a working directory and go to this directory Copy input files and executable to working directory Invoke your executable Copy output files to home directory or archive. A number of examples can be found at: https://subtrac.sara.nl/userdoc/wiki/huygens/usage/batch Copy-paste them and adapt them to your situation.
LoadLeveler Job example #!/bin/bash # @ node = 2 # @ tasks_per_node = 32 # @ notification = never # @ input = /dev/null # @ output = out.$(jobid) # @ error = err.$(jobid) # @ wall_clock_limit = 1:10:20 # @ job_type = parallel # @ network.mpi = sn_all,not_shared,us # @ queue # Here the shell script starts. cd $HOME/workdir./my_program
LoadLeveler Commands Intrinsic LoadLeveler commands llsubmit: submit a job llcancel: cancel a job llq: query LoadLeveler on job status llstatus: provide status of nodes Additional SARA scripts (to be found in /usr/sara/bin), providing extra views: llstx: give node occupation llstc: give class configuration of Huygens lljobs: alternative for llq
LoadLeveler Batch classes A Job is queued in a class by LoadLeveler (not in job script), based on user specified node usage, wall clock limit and number of nodes #@ node_usage = shared/not_shared #@ wallclock_limit = hh:mm:ss #@ nodes = 4 To allow LoadLeveler scheduler maximum freedom only a few classes are defined: (shared, short, normal, special). Use llstc to view the current configuration in classes:
LoadLeveler Batch classes 01 02 03 04 05 06 07 08 09 -----------------------------------------------------------------------------------------------------------------------------------6001 13 6013 13 6025 13 6037 13 6049 13 6061 13 6073 13 6085 13 6097 13 6002 13 6014 13 6026 13 6038 13 6050 13 6062 13 6074 13 6086 13 6098 13 6003 35 6015 13 6027 13 6039 13 6051 13 6063 13 6075 13 6087 13 6099 13 6004 35 6016 13 6028 13 6040 13 6052 13 6064 13 6076 13 6088 13 6100 13 6005 13 6017 2 6029 2 6041 13 6053 13 6065 2 6077 13 6089 13 6101 13 6006 13 6018 13 6030 13 6042 13 6054 13 6066 13 6078 13 6090 13 6102 13 6007 13 6019 13 6031 13 6043 13 6055 13 6067 13 6079 13 6091 13 6103 13 6008 35 6020 13 6032 13 6044 13 6056 13 6068 13 6080 13 6092 13 6104 13 6009 13 6021 13 6033 13 6045 13 6057 13 6069 13 6081 13 6093 13 6010 13 6022 13 6034 13 6046 13 6058 13 6070 13 6082 13 6094 13 6011 13 6023 13 6035 13 6047 13 6059 13 6071 13 6083 13 6095 13 6012 6024 4 6036 13 6048 13 6060 13 6072 4 6084 13 6096 13 -----------------------------------------------------------------------------------------------------------------------------------0 = unknown 1 = normal 2 = shared 3 = short 4 = spec_sys 5 = spec_training
LoadLeveler Scheduling policy Users can submit an unlimited number of jobs at one time. When a job is submitted it gets status: Queued (Eligible for scheduling) Not Queued (Not eligible for scheduling). Jobs are executed according to scheduling policy: First in - First out BUT: Only jobs in queued status are considered Timestamp when job acquires queued status is considered to be submission time. Number of jobs in queued status is limited Backfill mechanism applies
LoadLeveler Scheduling policy Configurable limits (by LoadLeveler administrator): Maxqueued: maximum number of jobs per user that can have queued status (default 8). Maxrunning: maximum number of jobs per user that can be running simultaneously (default 8). Maximum wallclock limit: (currently 5 days). Maximum number of nodes: (currently 80 in normal production mode).
LoadLeveler Submitfilter When you submit a job it is parsed by a submit filter (Huygens specific). It invokes the budget check script: Checks the remaining budget Computes the number of PNU s Of all jobs that have finished the same day, All jobs that are still in the queue It determines the class from the wallclock limit, number of nodes and node_usage. It checks whether or not some keywords are set. If some keywords are not set it sets them to a (sensible) default value. Defaults can be overruled by #@ skip_rset
Accounting Getting access to Huygens Accounts and Logins Budget and jobcost
Accounting Getting access to Huygens Take a look at the SARA website: https://subtrac.sara.nl/userdoc/wiki/huygens/access Proposal to NWO Filling in the forms in IRIS Peer review process Approval from NWO, What next? Granting letter (from NWO) A copy to SARA Acceptance letter (from NWO) Fill it in and return it to NWO. User form (download from SARA website) Fill in and send it to SARA. Usage agreement (download from SARA website) each user should fill this in, sign it and send it to SARA.
Accounting Accounts and Logins After receiving the proper forms SARA will create an account and login for you. Account: administrative entity to keep track of used budget Owner of an account is the PI (Principal Investigator) who submitted the project proposal to NWO. A project can have one or more accounts associated with it Each account can have several logins coupled to it. Duration of an account is 1 year (expiration date set by NWO) Login: combination of username + password and environment to give physical access to Huygens. Logins are STRICTLY PERSONAL. A login is at any time associated with one and only one account Logins can be moved from one account to another
Accounting Account expiration Monthly warnings will be sent (to PI!!!) that account will expire, starting 3 months before expiration date. Extension of an account is possible (contact NWO). Asking for extra time (budget will remain) Asking for extra budget (will be added to remaining budget) Submit a continuation proposal for extension of the same project (budget will be reset to new value). Submit a completely new proposal with new accounts (logins can be moved to new account). After expiration date the Account will be blocked Login to Huygens is denied You will be asked to give SARA permission to remove the login and all data associated with it from the system If you don t respond we will first seek permission of the Account Owner to remove everything. If still no response we will remove everything after a grace period of 6 months (in Usage Agreement).
Accounting Budget and jobcost Budget: If project proposal is accepted a budget is assigned to the accounts. Budget is expressed in PNU (Processor Node Uur) or equivalently in SBU (System Billing Units). 1 PNU = the use of 1 core for 1 hour on Huygens. Jobcost (Shared nodes) Jobcost based on cpu time Jobcost (Dedicated nodes) Jobcost based on wallclock time: You always pay for a complete node. Using 1 node for 1 hour will cost you 32 PNU (regardless of how many processes are running on the node). Capacity of Huygens: Gross: ~ 28M PNU/year. Nett: ~ 24M PNU/year (2M/month)
Accounting Budget and jobcost For overview of jobcosts use the command accuse Gives consumed budget per day or per month. For overview of budget use the command accinfo Information about initial, consumed and remaining budget. Gives contact information (e-mail address of account owner) Gives list of logins associated with the account.
Accounting Budget and jobcost Account : sscpwimr (HUYGENS2) Customer : (10305) Klant voor subinstelling 10305 Email : wimr@sara.nl Initial budget ; 60944:13 Used budget ; 17659:03 Remaining budget ; 43285:10 Creation date ; 2007-11-01 Last modified ; 2010-04-29 Valid until ; 2011-05-31. User ID(s) linked to this account: User Group -------------wimr ANY wimr2 ANY
Accounting Budget and jobcost Accounting information Jobinfo is kept by Loadleveler in a temporary file After the job finishes: Correct finish: the temporary file is added to a history file. Loadleveler crash: the temporary file is discarded Job restarted by system: temporary file is discarded Once a day (during the night): accounting information is extracted from this history file and added to the accounting database. The remaining budget is computed: If this is negative your account will be blocked. Budget check: To avoid that you will overtax your budget we introduce the budgetcheck, that will run at submission time and at job start. When remaining budget is not sufficient, your job will be refused.
User Support Expert user support (help desk, online documentation, (grid-) tutorials, courses) 7 full time HPC consultants (all computational scientists) for both Huygens and Lisa Optimization, parallelization/hyperscaling of user codes Benchmarking Visualization and data analysis of complex data sets Grid support Gridification of applications Grid-enabling of (local) resources Grid tutorials, education of traditional HPC users to use grid infrastructure Support for Grand-Challenge applications e-science support
Hands-On Contents Download necessary files locally Install user tools (Windows users only) Copy files to Huygens Login to Huygens Compile Molden (a comp. Chemistry tool) Look at input file with Molden Submit a job (geometry optimization) wait for the result... Analyze the result Copy back output/results locally
Hands-On Download Download the material from ftp://ftp.sara.nl/pub/outgoing/usinghuygens It includes: molden5.0.tar.gz Molecular Visualization Tool molecule.job, molecule.zmat Input for example For Windows users additionally: putty-0.62-installer.exe winscp437setup.exe Xming-6-9-0-31-setup.exe For Windows users: Install the three packages (mentioned above)
Hands-On Copy files to Huygens Mac & Linux users: Open a Terminal (Linux) or X11 (Mac) Go to the directory where you downloaded files Type: scp molecule.* molden5.0.tar.gz sdemonnn@huygens.sara.nl: where nnn is your demo number For Windows users: Start WinSCP Create New and fill in: Host name: huygens.sara.nl User name: sdemonnn Password: ******* Look up downloaded files and copy them to Huygens
Hands-On Login to Huygens Mac & Linux users: Open a Terminal (Linux) or X11 (Mac) Type: ssh -X sdemonnn@huygens.sara.nl where nnn is your demo number For Windows users: Start Xming (if not yet started system tray) Start PuTTY Host Name: huygens.sara.nl Click on Connection/SSH/X11 Check under X11 forwarding Enable X11 forwarding Click Open User your sdemonnn username and password to login
Hands-On Molden All users: Extract Molden tarball: tar zxf molden5.0.tar.gz Go into Molden directory: cd molden5.0 Modify makefile: gedit makefile change CC = cc into CC = gcc Make the binary: make Move the resulting binary to ~/bin: mv molden../bin Go back to home directory: cd.. Have a look at the molecule: molden molecule.zmat
Hands-On Inspect job (1) All users: Edit job script: gedit molecule.job
Hands-On Inspect job (2) LoadLeveler instructions: #!/bin/bash # @ node = 1 # @ tasks_per_node = 32 # @ input = /dev/null # @ output = molecule.$(jobid).out # @ error = molecule.$(jobid).err # @ wall_clock_limit = 1:00:00 # # @ notification = never # # @ job_type = parallel # @ network.mpi = sn_all,not_shared,us # # @ ll_res_id = p6012.603.r # @ queue
Hands-On Inspect job (3) Shell commands: STARTDIR=`pwd` echo "%NProcShared = 32" > $TMPDIR/molecule.inp echo "#RHF/3-21G Opt" >> $TMPDIR/molecule.inp echo "" >> $TMPDIR/molecule.inp echo "My molecule" >> $TMPDIR/molecule.inp echo "" >> $TMPDIR/molecule.inp echo "0,1" >> $TMPDIR/molecule.inp cat molecule.zmat >> $TMPDIR/molecule.inp cd $TMPDIR module load gaussian/09 g09 < molecule.inp
Hands-On Submit job All users: Submit job: llsubmit molecule.job Inspect status of your job: llq -u sdemonnn Once running, inspect ouputfile: tail -f molecule.<jobid>.out fill in job_id Once finished, analyze outputfile: molden molecule.<jobid>.out In Molden, press Movie See how benzene becomes flat and hexagonal!
Thank you for listening!