NEC HPC-Linux-Cluster Hardware configuration: 4 Front-end servers: each with SandyBridge-EP processors: 16 cores per node 128 GB memory 134 compute nodes: 112 nodes with SandyBridge-EP processors (16 cores per node) and 128 GB memory 4 nodes with SandyBridge-EP processors (16 cores per node) and 256 GB memory 18 nodes with Intel Haswell processors (24 cores per node) and 128 GB memory (at the moment only available for members of the excellence cluster Future Ocean and Geomar-Users) all compute nodes are connected with FDR-Infiniband network all nodes have access to a 1.5 PB global fast parallel filesystem (ScaTeFS) Access to the NEC-Linux cluster: login to the front-ends: ssh nesh-fe.rz.uni-kiel.de -l username resp. ssh -X nesh-fe.rz.unikiel.de -l username (if you need X-forwarding, graphical interface) the name nesh-fe is an alias for the login-servers nesh-fe1 to nesh-fe3, i.e. with the name nesh-fe you will be automatically redirected to one of the login nodes Operating system: Red Hat Linux (6.4); bash is standard shell Available file systems: each user has access to four different file systems: home-directory available via the environment variable $HOME path: /sfs/fs5/home-sh/username (CAU-Users) path: /sfs/fs6/geomar-sh/username (Geomar-Users) files stored in this directory have an unlimited life time will be backed up daily is available on all nodes only 32 TB for all university users and 32 TB for all Geomar-Users; no individual quota at the moment slower access time compared to the $WORK-directory work-directory available via the environment variable $WORK 9 different directories: please check your $WORK variable for the absolute pathname with the command echo $WORK is available on all nodes no backup of this directory unlimited life-time
no individual quota at the moment faster access time than to the $HOME directory local disk space on each node size: 500 TB on each cluster-node tape_cache-directory available via the environment variable $TAPE_CACHE access to our tape-library path: /nfs/tape_cache/username for files not needed for current calculations only available on the login node and in the batch class feque please store if possible only larger tar-files on this directory (max. size of one file: 500 GB) slow access time --> please don't work directly with files stored on the tape-library; please copy them first back to your $WORK-directory please don't use the rsync-command for copying and synchronizing files and directories to or from your your $TAPE_CACHE directory Available compilers: gnu-compiler: version 4.4.7 in the standard search path: gcc, gfortran and g++ newer versions available via modules module load gcc_4.8.2 (version 4.8.2) module load gcc_5.1.0 (version 5.1.0) Intel-compiler: version 14.0.0 and version 15.0.3.: ifort, icc and icpc please use the following command to set the correct environment variables before using the Intel-compiler: module load intel (version 14.0.0) module load intel15.0.3 (version 15.0.3.) MPI implementation for parallel programs: Intel-MPI please use the following command to set the correct environment variables before compiling or running an MPI-program: module load intelmpi compiling parallel MPI programs: gnu-compiler: mpif90, mpicc or mpicxx Intel-Compiler: mpiifort, mpiicc, mpiicpc For using more than one node for a parallel calculation, the ssh access must be configured for password-less access between the batch nodes. 1. To achieve this, the following steps are necessary: 1. type the command ssh-keygen -t dsa (confirm prompted questions only with enter-key) 2. copying the file $HOME/.ssh/id_dsa.pub into the file $HOME/.ssh/authorized_keys
Environment Modules We use on our NEC HPC-system the module concept to manage the environments for using different software packages, libraries and compiler versions. With the modules approach it is no longer necessary to explicitly specify paths for different software packages and it is easy to switch between different versions of the same software package. The main commands related to the environment modules are: module avail lists all available modules module load name adds the module named name to your environemt module unload name removes the module name and clears all corresponding settings module list lists all currently loaded modules module show name shows all settings which are performed by the module name man module provides details about the module command an all its subcommands Available libraries (only a selection): Intel MKL-library HDF5 library netcdf library Available software (only a selection): matlab R (version 3.1.1; no module load command necessary) perl (version 5.10.1) python: versions 2.6.6, 2.7 and 3.3.6 trinity samtools cufflinks Batch system and batch queues: please use for interactive work only the login node nesh-fe for longer calculations the following batch classes are available on the NEC cluster: clexpress: max. walltime 1 hour; 2 nodes available clmedium: max. walltime 48 hours (76 nodes available at the moment) cllong: max. walltime 100 hours at the moment (40 nodes available at the moment) clbigmem: max. walltime 100 hours; 4nodes with up to 256 GB memory are available clfocean: max. walltime 100 hours (4 nodes each with 16 cores and 128 GB memory) extra authorisation is necessary for this queue clfo2: max. walltime 100 hours (18 nodes each with 24 cores and 128 GB memory) extra authorisation is necessary for this queue feque: one node available; e.g. for transferring data to the tape_library via batch
Commands for submitting and monitoring batch jobs: qsub nec_cluster.nqs send the batch script with the name nec_cluster.nqs to the batchsystem qstatall information about all waiting and running batch jobs on the whole system (Linux cluster and vector nodes) qstatcl and qstatace information about all batch jobs on the Linux cluster nodes (qstatcl) or vector nodes (qstatace) qstat information about your own batchjobs qdel jobid: command to delete one of your jobs qcl: overview of the currently available free nodes in the different batch queues Example batch script for a serial calculation: #!/bin/bash #PBS -q clmedium --> name of the batch class #PBS -l cpunum_job= 1 --> number of cores per node (here 1 core) #PBS -b 1 ---> number of nodes (here 1 node) #PBS -l cputim_job=32:00::00 --> CPU-time (= at least corenumber * walltime) #PBS -l elapstim_req=2:00:00 ---> walltime requested #PBS -l memsz_job=10gb --> memory request per node #PBS -j o --> redirect standard and error output into the same file #PBS -o clustertest.out ---> name of the standard output #PBS -m bea --> specifies that the batch system send an email notification when the job begins (b), ends (e), or aborrs (a). #PBS -M mustermann@xxx.uni-kiel.de --> send status information to this email adress cd $PBS_O_WORKDIR. /opt/modules/modules/3.2.6/init/bash --> initialisation of the module-concept module load intel --> set the correct environment variables for the intel-compiler module load trinity --> setting environment variables for special software (here e.g. trinity)./name_of_the executable --> starting a serial program explanation for the line. /opt/modules/modules/3.2.6/init/bash: Be aware of the fact that there is a space between the dot and /opt
Example batch script for a parallel calculation: this example batch script runs on four nodes each with 16 cores #!/bin/bash #PBS -q clmedium --> name of the batch class #PBS -l cpunum_job=16 --> number of cores per node #PBS -b 4 ---> number of nodes (here 4 nodes each with 16 cores) #PBS -l cputim_job=32:00::00 --> CPU-time (= at least corenumber * walltime) #PBS -l elapstim_req=2:00:00 ---> walltime requested #PBS -l memsz_job=10gb --> memory request per node #PBS -T intmpi ---> line is necessary for running MPI-programs #PBS -j o --> redirect standard and error output into the same file #PBS -o clustertest.out ---> name of the standard output cd $PBS_O_WORKDIR. /opt/modules/modules/3.2.6/init/bash --> initialisation of the module-concept module load intel intelmpi --> set the correct environment variables for the intel-compiler moduleload trinity --> setting environment variables for special software (here e.g. trinity) mpirun $NQSII_MPIOPTS -np 64./prog.exe --> starting a parallel program explanation for the line. /opt/modules/modules/3.2.6/init/bash: Be aware of the fact that there is a space between the dot and /opt