Vital-IT Users Training: HPC in Life Sciences

Vital-IT Users Training: HPC in Life Sciences Vital-IT Group, Lausanne Contact: projects@vital-it.ch Status: 29 January 2015

Objectives of this course Obtain basic knowledge on high throughput and high performance computing Learn how to use computer clusters at Vital-IT Tips and tricks for effective and efficient cluster usage 2, HPC in Life Sciences

Outline 1. Motivation 2. Background on Scientific Computing 3. Practical usage of life science HPC infrastructure 4. Vital-IT: Different storage systems and guidelines 3, HPC in Life Sciences

Motivation Set up a desktop machine (PC) with bioinformatics software Run e.g. tophat, BLAST, HMMER, velvet etc. on a small data set Possible? How long does it take? Run it against a larger data set, multiple genomes etc. Possible? How long does it take? Are we happy with the software/ hardware environment? Performance? Storage space? Maintenance? etc. 4, HPC in Life Sciences

Bioinformatics Competence Center http://www.vital-it.ch/ 5, HPC in Life Sciences

Missions To support research groups in biology and medicine To maintain and develop algorithms in life science To teach and liaise with embedded bioinformaticians (education in algorithm development and use of high performance computing environment) To keep active research inside Vital-IT, mainly through collaborations with partners To operate high productivity computational and storage resources http://www.vital-it.ch/about/activities.php 6, HPC in Life Sciences

Vital-IT Team http://www.vital-it.ch/about/team.php 7, HPC in Life Sciences

Vital-IT and Partner Resources in Switzerland Arc lemanic (3 clusters managed by Vital-IT team) UNIL, EPFL, UNIGE (>3500 processing cores) One login for all clusters; LSF for job submission Production usage is based on project approval! Linux CentOS 6.4, parallel file system, large storage archive University of Berne Operated by Uni Berne Informatikdienste UBELIX (http://www.ubelix.unibe.ch/) ~1200 cores; SGE for jobs submission Vital-IT software + databases are available in collaboration with Vital-IT 8, HPC in Life Sciences

Vital-IT and Partner Resources in Switzerland EPFL UNIL UNIGE 9, HPC in Life Sciences

Resources on the Vital-IT Web Site CPU cores 10, HPC in Life Sciences

Outline 1. Motivation 2. Background on Scientific Computing Compute Cluster Parallelism 3. Practical usage of life science HPC infrastructure 4. Vital-IT: Different storage systems and guidelines 11, HPC in Life Sciences

A typical compute cluster Usually, a cluster consists of Front-end machine where users log in and submit jobs Compute nodes (also called worker nodes or hosts) where the actual computation takes place Typically, one is not allowed to log into compute nodes Specialized software for submitting jobs to compute nodes, also referred to as Local Resource Management System (LRMS) or batch system LSF (Load Sharing Facility) Vital-IT (UNIL/EPFL/UNIGE) SGE (Sun/Oracle Grid Engine) Uni Berne (UBELIX) etc. 12, HPC in Life Sciences

A typical compute cluster (cont.) LSF: UNIL, EPFL, UNIGE login node (front-end) Local Resource Management System SGE: Uni Berne host host compute nodes host Several processing cores (CPUs) per host. A host is identified by a single host name. 13, HPC in Life Sciences

When should I use a cluster? An application runs too long on a single PC and takes too much CPU Just run the single job on a cluster The problem (input data) can be split into pieces which can be executed in parallel Data splitting and final merging Parameter sweep: only some input parameters are changed but the same application is executed several times 14, HPC in Life Sciences

Bioinformatics examples NGS, genome analysis (data intensive) tophat, bowtie, velvet, abyss, Solexa GA-Pipeline etc. Sequence search and alignment BLAST, HMMER etc. Molecular dynamics NAMD, GROMACS, CHARMM etc. Computational phylogeny Phylip, PAML, BEAST, RAxML etc. Protein identification X!Tandem, Mascot etc. 15, HPC in Life Sciences

Outline 1. Motivation 2. Background on Scientific Computing Compute Cluster Parallelism 3. Practical usage of life science HPC infrastructure 4. Vital-IT: Different storage systems and guidelines 16, HPC in Life Sciences

High Performance/Throughput Computing: run things in parallel Embarrassingly parallel Massively parallel Jobs run independently Multiple results are produced Communication, synchronisation classic HPC, uses MPI on supercomputers 17, HPC in Life Sciences

Outline 1. Motivation 2. Background on Scientific Computing 3. Practical usage of life science HPC infrastructure - Vital-IT UNIL/EPFL/UNIGE infrastructure overview - Concrete cluster usage 4. Vital-IT: Different storage systems and guidelines 18, HPC in Life Sciences

Using the Vital-IT Cluster at UNIL In fact, there are two clusters! Production cluster (prd.vital-it.ch front-end / login node) More than 2000 CPUs on more than 140 compute nodes This login node is visible on the Internet (register your IP address to see this machine) Development cluster (dev.vital-it.ch front-end / login node) 2 compute nodes This cluster is not visible on the Internet Hardware environment is rather homogeneous Based on Intel s Xeon architecture http://www.vital-it.ch/prdpub/ganglia-webfrontend/ No computation/extensive CPU usage on prd/dev frontends! If you need to compile and test code, please do it on dev. 19, HPC in Life Sciences

How to login to Vital-IT s clusters UNIL (Uni Lausanne) ssh prd.vital-it.ch ssh dev.vital-it.ch EPFL ssh frt.el.vital-it.ch UNIGE (Uni Geneva) ssh frt.ug.vital-it.ch 20, HPC in Life Sciences

Vital-IT Production Cluster Status Example http://www.vital-it.ch/prdpub/ganglia-webfrontend/ 21, HPC in Life Sciences

How do we get >2000 CPUs on the UNIL/Prod Cluster (~150 hosts)? Host name Archit. CPUs Speed RAM total cpt037-cpt100 Intel 8 3 GHz 16 GB 64 * 8 = 512 cpt133-cpt148 Intel 4 3 GHz 16 GB 16 * 4 = 64 cpt149-164 Intel 4 3 GHz 8 GB 16 * 4 = 64 (165,166,181-184,186)* AMD 64 2.3 GHz 512 GB 7 * 64 = 448 cpt187, 188 AMD 48 2.3 GHz 256 GB 2 * 48 = 96 cpt197-228 Intel 8 2.8 GHz 16 GB 32 * 8 = 256 dee-cpt01-08 Intel 16 2.5 GHz 24 GB 8*16 = 128 Not all CPUs are available all the time! 22, HPC in Life Sciences Need clever software for selecting suitable compute node. (Local Resource Management System) *Priority/reserved for some UNIL groups

Large memory machines rserv.vital-it.ch AMD, 8 cores, 64 GB RAM rserv01.vital-it.ch AMD, 64 cores, 256 GB RAM Usage policies: Dedicated to the memory-intensive jobs LSF cannot be used to submit jobs to this machine Use direct ssh connections to the machine (from prd) Other machines with 256 GB and 512 GB RAM restricted access Intel and AMD Opteron (incl. GPUs for specific projects) EPFL/UNIGE clusters: min. 128 GB RAM 23, HPC in Life Sciences

How to use the cluster Linux terminal interface UNIX knowledge is essential ssh access to front-end only No need to select a compute node yourself We have LSF (Load Sharing Facility) that Knows about all compute nodes Accepts user jobs Submits jobs to a compute node Knows about status of jobs 24, HPC in Life Sciences

LSF (Load Sharing Facility) login node (front-end) prd.vital.it (LSF client) LSF Server cpt037 compute nodes cpt141 compute nodes cpt191 25, HPC in Life Sciences

Jobs and Queues Every program to be executed on a cluster needs to be defined as a job with a job description: Executable, input data, output data, etc. Job characteristics (CPU, memory, architecture etc.) Priorities and run time limitations Queues are used to classify jobs Different queues for different priorities/running times normal, long, priority, etc. Job states: pending, running, suspended, done, exit (usually refers to an error) 26, HPC in Life Sciences

Vital-IT UNIL s main LSF queues in detail queue$name$ (me$limit$ max.$jobs$ simultan.$ max.$cpus$per$ job$ Default$RAM$ in$gb$ $ priority$ normal' default(queue( 24'hours' 150' 64' 2' medium' long' 10'days' 64' 16' 2' low' priority' 60'min' (interact.( jobs)( keep(it( low ( moderate'' 2' high' Beyond'this':me' limit,'jobs'might' be'killed.' 27, HPC in Life Sciences That s'the'max.'number' of'jobs'a'single'user''can' ac:vely'run'at'any'given' :me.' Parallel' execu:ons'per' job.' Each'CPU'is'shared' (poten:ally'with'other'users).' One'cannot'reserve'a'single' CPU'for'a'single'job.' Jobs'with'higher'priority'are' treated'first'(preemp:ve' jobs)'

LSF queues at EPFL and UNIGE Similar queues/setup but machines have more cores and more memory EPFL 16 hosts, 48 CPUs (cores) each Queue$ RAM$ default$ RAM$$ max$ normal' 2'GB' 128'GB' long' 2'GB' 128'GB' priority' 2'GB' 64'GB' UNIGE 19 hosts, 48 CPUs (cores) each Queue$ RAM$ default$ RAM$$ max$ normal' 2'GB' 238'GB' long' 2'GB' 57'GB' priority' 2'GB' 57'GB' interac:ve' 2'GB' 238'GB' 28, HPC in Life Sciences

Usage hints basic overview Which queue should I use? Try to estimate overall processing time Potentially, test with priority queue for short and interactive jobs Where to write data/results? /scratch/cluster/[daily weekly monthly]/userid OR /scratch/local/[daily weekly]/userid Please don t write to /home/ if job runs on cluster! Directories are shared between all compute nodes Shared file systems (except /scratch/local!) Where to archive (long term storage)? There is an archive system More'details'on' storage'will'be' given'later' 29, HPC in Life Sciences

Which version of the software + database? You always need to know which software version and database version you use Version might change over time! Might make all your experiments invalid! Always check version before you start experiments Don t rely on default installations blindly Verify again and again! $ bowtie2 --version version 2.1.0 30, HPC in Life Sciences

Applications (bioinformatics software tools) http://www.vital-it.ch/support/tools.html Many applications are installed Specific version needs to be selected explicitly e.g. bwa-0.7.5a 31, HPC in Life Sciences

Accessing Vital-IT Software General convention: /software/<category>/<package>/<version>/ <category> scientific/technical category such as UHTS, Phylogeny, LC- MS-MS, Development, etc. <package> is the general name of the software package <version> version of the package incl. different compiler/architecture names if needed. /software/uhts/aligner/bowtie2/2.2.1 /software/uhts/quality_control/fastqc/0.10.1 /software/phylogeny/paml/4.5 /software/development/java_jre/1.8.0_05 /software/r/3.0.2 /software/bin/gcc 32, HPC in Life Sciences

Activate package: module avail add rm Check what is available (not necessarily activated!) module avail Activate (add) a specific software tool to PATH module add UTHS/Quality_control/fastqc/0.10.0 Check what is activated module list Deactivate (remove) a software tool module rm UHTS/Quality_control/fastqc/0.10.0 33, HPC in Life Sciences

Accessing Vital-IT Software [vioannid@frt ~]$ vit_soft case'insensi:ve'search' Missing argument Tool to retrieve information about Vital-IT software (version: 0.0.5) Usage: vit_soft -search=<uvw> vit_soft -package=<xyz> vit_soft -module=<xyz> vit_soft -binary=<abc> vit_soft -latest vit_soft -help Text search for packages (c.i.) Get package information Get module command for a package Search package providing this binary Get latest installed packages More details 34, HPC in Life Sciences

Accessing Vital-IT Software vit_soft -s tie VitalIT-Utilities-1.0.1-1.x86_64 perl-tie-toobject-0.03-1.x86_64 perl-tie-ixhash-1.22-1.el5.centos.noarch perl-html-entities-numbered-0.04-1.x86_64 bowtie2-2.2.1-1.x86_64 bowtie2-2.1.0-1.x86_64 bowtie2-2.0.0beta6-2.x86_64 bowtie-0.12.9-1.x86_64 Same'as:' vit_sow'' search':e' vit_soft -m bowtie2 module add UHTS/Aligner/bowtie2/2.2.1 module add UHTS/Aligner/bowtie2/2.1.0 module add UHTS/Aligner/bowtie2/2.0.0beta6 Same'as:' vit_sow' module'bow:e2' 35, HPC in Life Sciences

Outline 1. Motivation 2. Background on Scientific Computing 3. Practical usage of life science HPC infrastructure - Vital-IT UNIL/EPFL/UNIGE infrastructure overview - Concrete cluster usage - Using UBELIX + Vital-IT software 4. Vital-IT: Different storage systems and guidelines 36, HPC in Life Sciences

LSF command line client Interact with LSF queue using the following basic commands: bsub submit job bjobs obtain job status bkill kill job Other useful commands bswitch move job to another queue bhist historical information on job bpeek display std. output/error of unfinished job bqueues information about available queues 37, HPC in Life Sciences

Basic LSF example (1) Example command we want to execute: blastall -p blastp -d "swiss" -i p123456.seq Write a shell script (job description) blast.sh: #!/bin/bash blast.sh #BSUB -L /bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt standard'output(is'redirected' to'a'file'indicated'by'yo' standard'error'indicated'by'ye' blastall -p blastp -d "swiss" -i p123456.seq 38, HPC in Life Sciences

Basic LSF example (2) ''Submit'job' ' Note'the' < 'sign!' bsub <./blast.sh Job <870965> is submitted to queue <normal>. ''Check'status' ' Job'ID'that'uniquely' iden:fies'a'job' bjobs 870965 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 870965 user007 RUN normal frt cpt204 *23456.seq Nov 16 16:40 bjobs No unfinished job found ls blast-error.txt blast-output.txt Result'is'ready'if'job'is'no' longer'displayed.' Note:'bjobs'without'job' ID'displays'informa:on'on' all'of'your'ac:ve'jobs.' ' 39, HPC in Life Sciences

Basic LSF example (3) bjobs 870965 If'job'ID'is'used'explicitly,'also'the' DONE'status'is'shown.' JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 870965 user007 DONE normal frt cpt204 *23456.seq Nov 16 16:40 bjobs a Info'about'all'jobs'(running'and'finished).' Only'available'for'a'few'hours!' JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 884535 user007 DONE normal frt cpt129 *;hostname Nov 17 11:17 884538 user007 DONE normal frt cpt129 *;hostname Nov 17 11:18 884568 user007 DONE normal frt cpt014 *simple.sh Nov 17 11:22 884595 user007 DONE normal frt cpt194 *23456.seq Nov 17 11:28 884633 user007 EXIT normal frt cpt014 *stderr.sh Nov 17 11:34 884657 user007 DONE normal frt cpt005 *stderr.sh Nov 17 11:37 40, HPC in Life Sciences

Where to write results, store files Location for input files and results (output files): /scratch/ Location for scripts /home/<userid> or /scratch/ 41, HPC in Life Sciences

How do we apply that to our basic example? Script to launch blastall /home/user007/blast.sh Create directory in /scratch mkdir /scratch/cluster/weekly/user007 Change working directory to /scratch cd /scratch/cluster/weekly/user007 Launch job with reference to blast.sh in home directory bsub < /home/user007/blast.sh 42, HPC in Life Sciences Please'do'that'for'(all)'your'jobs!' Other'op:ons'to'write'into'/scratch?'

LSF example define a job name Give a job a meaningful name #!/bin/bash #BSUB -L /bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt #BSUB -J blastp blastall -p blastp -d "swiss" -i p123456.seq blast-with-jobname.sh bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 885321 user007 DONE normal frt cpt129 blastp Nov 17 14:00 43, HPC in Life Sciences

LSF example use job ID in output file name Distinguish jobs and files #!/bin/bash #BSUB -L /bin/bash #BSUB -o blast-%j-output.txt #BSUB -e blast-%j-error.txt blastall -p blastp -d "swiss" -i p123456.seq blast-with-jobid.sh bsub <./blast-with-jobid.sh Job <885626> is submitted to default queue <normal>. $ls *.txt blast-885626-error.txt blast-885626-output.txt 44, HPC in Life Sciences

LSF example e-mail notification Receive an e-mail when job has finished Includes CPU usage information (timing) etc. #!/bin/bash #BSUB -L /bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt #BSUB -u MyEmail@example.org #BSUB -N blastall -p blastp -d "swiss" -i p123456.seq blast-with-email.sh See example e-mail on next slide 45, HPC in Life Sciences

. Your job looked like: ------------------------------------------------------------ # LSBATCH: User input #!/bin/bash eamail$is$sent$by$lsf$server$ #BSUB -L /bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt #BSUB u MyEmail@example.org #BSUB -N Job'descrip:on' blastall -p blastp -d "swiss" -i p123456.seq ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 2.44 sec. Max Memory : 1 MB Max Swap : 15 MB Job'status' Resource'usage' (performance' informa:on)' Max Processes : 1 Max Threads : 1 Read file <blast-output.txt> for stdout output of this job. Read file <blast-error.txt> for stderr output of this job. 46, HPC in Life Sciences Informa:on'on'output'and' error'files.'

Interactive Job Job output is displayed on the terminal rather than written into a file Gives impression that job is executed on the local machine Waits until job has finished echo "The hostname is... " hostname bsub -I <./simple.sh Job <886075> is submitted to default queue <normal>. <<Waiting for dispatch...>> <<Starting on cpt194>> The hostname is... cpt194 simple.sh 47, HPC in Life Sciences

Interactive Job with X11-forwarding Jobs with a graphical display can be used in interactive mode graphical interface can be displayed (forwarded) on log-in node (ssh X) #!/bin/bash #BSUB -L /bin/bash #BSUB -o output.txt #BSUB -e error.txt #BSUB -J interact #BSUB -XF xterm beast terminal'type' X11-forwarding.sh 48, HPC in Life Sciences

Processes, threads, multitasking A process is an executing - or running - program identified by a unique process identifier (PID). Looking closer, a process is some kind of entity - sometimes referred to as an "allocation unit" - that comprises several things, such as the running program, a reserved space in memory, temporary data, etc. A thread is a 'light weight' process - or "unit of execution" - that is contained in a process. It uses the same environment as the process it belongs to - PID, memory, data, etc. Each process has one or more threads but each thread belongs to one process only! 49, HPC in Life Sciences

Processes, threads, multitasking Long ago in computer age a computer contained one central processing unit CPU. One machine 1 CPU / 1 core On such a computer, only ONE process is active at a time. And for processes with several threads, only ONE thread is active at a time even if we get the impression that several threads run in parallel. Indeed, the CPU switches very fast between the threads - some CPU time is given to process one, then some CPU time is given to process two, etc. until the all the processes have completed - which gives the impression that they "kind of run in parallel". 1 process With 4 threads All the threads run sequentially. Thus, the total execution time is the sum of the time needed to complete all four threads individually. 50, HPC in Life Sciences

Processes, threads, multitasking Based on the fact that '1 thread -> 1 CPU', engineers have multiplied the CPU - or cores (*) to allow multiple threads to run simultaneously. One machine 4 CPUS / 4 cores This means also, that the more cores there are, the more threads can be executed at the same time - for real this time! (*) Cores and CPU are not exact synonyms. Actually, the execution parts of the CPU were duplicated and reorganized into 'cores'. Thus, a CPU can contain one or more cores - bi-core, quadri-core, etc. 1 process With 4 threads All the threads run at the same time - in parallel. Thus the total execution time is the time needed to complete one thread - assuming they take the same time to complete. 51, HPC in Life Sciences

Multi-threaded jobs (1 thread per core) By default, LSF reserves 1 core (CPU) per job Multi-threaded jobs need to request more cores on the same host (machine)! #!/bin/bash #BSUB -L /bin/bash #BSUB -o output.txt #BSUB -e error.txt #BSUB -n 4 #BSUB R "span[ptile=4]" 4'CPU'cores'(processors)' requested' 4'cores'on'the'same' host!' myprogram --threads=4 52, HPC in Life Sciences

Resource/memory requirements Job can explicitly require certain hardware resources such as RAM (main memory), swap space, etc. 2 GB RAM max per default! #!/bin/bash #BSUB -L /bin/bash #BSUB -o output.txt #BSUB -e error.txt #BSUB R "rusage[mem=4096]" #BSUB -M 4194304 hostname Required'memory'in'MB# (at'scheduling':me)' Required'memory'in'KB# (at'job'run':me)' mem-requirement.sh 53, HPC in Life Sciences

Cluster resources (lsload) temp. space /scratch/local RAM (main memory) 54, HPC in Life Sciences

Memory requirement and multi-core Request a multi-core job that needs lots of memory (e.g. 40 GB on one machine) #!/bin/bash #BSUB -L /bin/bash #BSUB -o output.txt #BSUB -e error.txt #BSUB -J mem-mult #BSUB -n 4 #BSUB -R "span[ptile=4]" #BSUB -R "rusage[mem=40000]" #BSUB -M 40194304./myprogram1 mem-multicore.sh 55, HPC in Life Sciences

Request/use local disk on compute node 10 GB at scheduling time (select) 6 GB at run time (rusage) Clean up directory yourself! #!/bin/bash #BSUB -L /bin/bash #BSUB -o output.txt #BSUB -e error.txt #BSUB -J localdisk #BSUB R "select[tmp>$((10*1024))]" #BSUB -R "rusage[tmp>$((6*1024))]" mkdir /scratch/local/daily/mydir [ ] rm rf /scratch/local/daily/mydir 56, HPC in Life Sciences

Check the availability of compute nodes lshosts -R "select[maxmem=400000]" HOST_NAME type model cpuf ncpus maxmem maxswp server RESOURCES cpt165 X86_64 Opteron8 60.0 64 524237M 31999M Yes () cpt166 X86_64 Opteron8 60.0 64 524237M 31999M Yes () cpt181 X86_64 Opteron8 60.0 64 524237M 31999M Yes () cpt182 X86_64 Opteron8 60.0 64 524237M 31999M Yes () cpt183 X86_64 Opteron8 60.0 64 524237M 31999M Yes () cpt184 X86_64 Opteron8 60.0 64 524237M 31999M Yes () cpt186 X86_64 Opteron8 60.0 64 524237M 31999M Yes () Memory'size' Installed'in'host' 57, HPC in Life Sciences

Big'memory'requirements' will'rapidly'decrease'the'list' of'poten:al'execu:on'hosts.' Check the availability of compute nodes lsload -R "select[maxmem=400000]" HOST_NAME status r15s r1m r15m ut pg ls I t tmp swp mem Cpt165 ok 0.0 0.0 0.0 0% 0.0 1 3998 1006G 31G 500G Cpt182 ok 23.3 22.8 23.4 37% 0.0 1 94 505G 31G 336G cpt186 ok 26.6 24.9 25.4 43% 0.0 0 24304 1004G 31G 305G cpt183 ok 26.8 25.3 25.4 41% 0.0 0 5592 1003G 31G 245G cpt184 ok 27.1 25.4 26.1 44% 0.0 0 5496 1004G 31G 318G cpt181 ok 28.2 26.5 26.4 44% 0.0 0 24304 1004G 31G 310G cpt166 ok 32.8 33.9 36.4 53% 0.0 0 2388 778G 30G 402G lsload -R "select[mem=400000]" Current'available' memory' HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem cpt165 ok 0.0 0.0 0.0 0% 0.0 1 3998 1006G 31G 500G cpt166 ok 36.7 34.2 36.4 54% 0.0 0 2388 778G 30G 401G 58, HPC in Life Sciences

Check the availability of compute nodes lshosts gives information about hosts hardware specs lsload gives information about hosts current available resources (maxmem, mem, tmp, status, swp, etc.) 59, HPC in Life Sciences

Status pending for resource-hungry jobs If job is pending for a long time, use option p or -l to get more info why it is pending: bjobs'yp'164498' JOBID'''USER''''STAT''QUEUE''''''FROM_HOST'''EXEC_HOST'''JOB_NAME''' SUBMIT_TIME' 164498''userid'PEND''normal'''''frt'''''''''''''''''''''testjob''''May''4'09:59' 'Closed'by'LSF'administrator:'15'hosts;' 'Not'specified'in'job'submission:'52'hosts;' 'Job'requirements'for'reserving'resource'(mem)'not'sa:sfied:'95' hosts;' 'Load'informa:on'unavailable:'3'hosts;' 'Just'started'a'job'recently:'2'hosts;' '' 60, HPC in Life Sciences

Job runs too long: move it to another queue By default, jobs in the normal queue can run for 24 hours If you see that it might need to run longer, move the job to the long queue bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 85613 testuse RUN normal frt.el.vita cpt16.el.vi lsftest Dec 7 12:10 bswitch long 85613 Job <85613> is switched to queue <long> 61, HPC in Life Sciences

Array Job: submit the same job multiple times Motivation/example: My simulation software mysim needs to be executed 20 times Submit a single job rather than 20 individual ones #!/bin/bash Index$in'job'array' #BSUB -L /bin/bash #BSUB -J array[1-20] #BSUB -o array-output-%j-%i.txt #BSUB -e array-error-%j-%i.txt array-job.sh run-mysim --seed=$lsb_jobindex Job'ID$ 62, HPC in Life Sciences

Job arrays (cont.) bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 832999 user007 RUN normal frt cpt130 array[16] Mar 8 11:55 832999 user007 RUN normal frt cpt101 array[9] Mar 8 11:55 832999 user007 RUN normal frt cpt119 array[2] Mar 8 11:55 832999 user007 RUN normal frt cpt127 array[4] Mar 8 11:55 832999 user007 RUN normal frt cpt106 array[3] Mar 8 11:55 832999 user007 RUN normal frt cpt131 array[1] Mar 8 11:55 832999 user007 RUN normal frt cpt140 array[10] Mar 8 11:55 832999 user007 RUN normal frt cpt138 array[12] Mar 8 11:55 832999 user007 RUN normal frt cpt139 array[15] Mar 8 11:55 Nota bene: The same program will be executed multiple times Be careful when you write output files: Need to be sub-job specific $LSB_JOBINDEX env. var can be used in script $LSB_JOBID for entire job array! 63, HPC in Life Sciences

Pitfalls with LSF (1) bsub./blast.sh If'you'forget'to'use' <,'the'job' output$is$sent$via$eamail!$ ' Correct'command:' bsub <./blast.sh Job <886355> is submitted to default queue <normal>. bjobs 886355 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 886355 hstocki DONE normal frt cpt187./blast.sh Nov 17 16:34 ls blast.sh p123456.seq Job'output'is'not'available'even'if'the'job' seems'to'have'finished'correctly.'' 64, HPC in Life Sciences

Pitfalls with LSF (2) #!/bin/bash #BSUB -L /bin/bash #BSUB -e blast-error.txt blastall -p blastp -d "swiss" -i p123456.seq blast-woo.sh bsub <./blast-woo.sh If'standard'output'is'not' men:oned,'it'will'not'be'created' in'file'system'but'sent'via'eymail' (with'delay)!'add:' ' #BSUB -o blast-output.txt ' 65, HPC in Life Sciences

Pitfalls with LSF (3) By default, email notifications are sent to mailbox connected to UNIX user id: userid@cptxxx.prdclst.vital-it.ch Need to explicitly mention different e-mail address YN'is'missing'to' explicitly'evoke' no:fica:on!' #!/bin/bash #BSUB -L /bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt #BSUB -u MyEmail@example.org blastall -p blastp -d "swiss" -i p123456.seq blast-with-email2.sh 66, HPC in Life Sciences

Further information LSF commands man pages (man bsub, man bjobs etc.) All commands http://www.vital-it.ch/support/lsf/print/lsf_command_ref.pdf Infrastructure, cluster status, software, support etc. http://www.vital-it.ch 67, HPC in Life Sciences

Vital-IT Helpdesk If you have project-related questions on how to use Vital-IT software etc., please send e-mail to: projects@vital-it.ch Try to be as specific as possible when asking a question/reporting a problem so it can be reproduced. Please specify: Machine/cluster Software package (command you used) + version Input data and directory where you worked Output/error you have received Copy/paste command used + full error message 68, HPC in Life Sciences

Summary: what can you expect from Vital- IT and the team? High performance cluster, i.e. hardware and software infrastructure Wide range of bioinformatics applications New applications can be added on request Expertise in several domains of biology, bioinformatics and computer science (IT) The Vital-IT team Can help you prepare and run your project on the cluster but cannot solve all questions related to all software applications! 69, HPC in Life Sciences

"New applications can be added on request" The Vital-IT team is making every effort to answer all requests as soon as possible. However: creation of accounts may take several days installation of new software takes 1-2 weeks (but can take more if the software is complex) Vital-IT expects you to help testing the application that you requested to install 70, HPC in Life Sciences

SGE (Sun/Oracle Grid Engine) login node (front-end) submit.unibe.ch SGE client SGE Server cnode01 compute nodes dnode03 compute nodes fnode02 72, HPC in Life Sciences

LSF vs SGE (UBELIX) main difference Task$ LSF$ SGE$ Submit'job' bsub' qsub' Check'status' bjobs' qstat' Cancel/Kill'job' bkill' qdel' 73, HPC in Life Sciences

LSF vs SGE (UBELIX) main difference LSF: bsub < blast.sh #!/bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt module add Blast/ncbi-blast/2.2.29+ blastall -p blastp -d "swiss" -i p123456.seq SGE: qsub sge-blast.sh #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd #$ -o blast-output.txt #$ -e blast-error.txt module load vital-it module add Blast/blast/2.2.26 blastall -p blastp -d "swiss" -i p123456.seq Star:ng'12'February'2015:' module'load 'is'now'mandatory!' 74, HPC in Life Sciences

Compute nodes on UBELIX Note the difference of architecture (Intel/AMD), number of cores, RAM etc. Total: 2288 processing cores ~8 TB RAM ~300 TB disk space (shared file system) disks mentioned above are local disks! 75, HPC in Life Sciences

Storage on UBELIX Not available from compute nodes! *2 TB per user, 15 TB per group may be extended by contacting grid-admin@id.unibe.ch ** quota can be requested when submitting job 76, HPC in Life Sciences

UBELIX SGE queues in detail queue$name$ (me$limit$ max.$jobs$ simultaneously$ max.$cpus$ per$job$ priority$ all.q' default(queue( 360'hours' ='15'days' 150' 8' medium' short.q' 1'hour' 30' 8' high' mpi.q' 360'hours' ='15'days' 200' 128' medium' Beyond'this':me' limit,'jobs'might' be'killed.' 77, HPC in Life Sciences That s'the'max.'number' of'jobs'a'single'user''can' ac:vely'run'at'any'given' :me'(limited'also'by' CPUs/job).' Parallel'execu:ons' per'job.' (MPI:'If'128'CPUs/ job,'only'1'job'is' possible.'if'64' CPUs/job,'max'3' jobs.)' Each'CPU'is'shared' (poten:ally'with'other'users).' One'cannot'reserve'a'single' CPU'for'a'single'job.' Jobs'with'higher'priority'are' treated'first'(preemp:ve' jobs)'

In summary: SGE command line client Interact with SGE queue using the following basic commands: qsub submit job qstat obtain job status (qacct) qdel delete/kill job UBELIX web site: hqp://www.ubelix.unibe.ch/' 78, HPC in Life Sciences

Before we start: let s set the environment Environment on cluster nodes is different to login node Need to explicitly set paths That is particularly required for certain Vital-IT applications and databases (/mnt/local/bin for applications) # Set PATH export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib" for i in /mnt/common/etc/profile.d/*.sh; do if [ -x $i ]; then. $i; fi; done 79, HPC in Life Sciences

Basic SGE example (1) Example command we want to execute: blastall -p blastp -d "swiss" -i p123456.seq Write a shell script (job description) sge-blast.sh sge-blast.sh #!/bin/bash # Set PATH [ ] (see previous slide) #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd CPU':me'requirement'(REQUIRED)$ memory'requirement'(required)$ current'working'directory' blastall p blastp -d "swiss" -i p123456.seq 80, HPC in Life Sciences

Basic SGE example (2) ''Submit'job' ' qsub blast.sh Your job 3754029 ( sge-blast.sh") has been submitted ''Check'status' ' Job'ID'that'uniquely' iden:fies'a'job' qstat job-id prior name user state submit/start at queue -------------------------------------------------------------------------------- 3754030 0.00000 sge-blast.sh userid qw 02/01/2012 11:56:05 qstat ls 3754030.err 3754030.out Result'is'ready'if'job'is' no'longer'displayed.' ' 81, HPC in Life Sciences

Basic SGE example (3) qacct j 870965 qname short.q hostname cnode02.ubelix.unibe.ch group groupid owner userid [ ] qsub_time Wed Feb 1 11:56:05 2012 start_time Wed Feb 1 11:56:18 2012 end_time Wed Feb 1 11:56:20 2012 granted_pe NONE slots 1 failed 0 exit_status 0 ru_wallclock 2 [ ] cpu 2.196 mem 0.428 io 0.000 iow 0.000 maxvmem 320.426M arid 82, HPC in Life Sciences undefined

SGE example define a job name Give a job a meaningful name #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd #$ -N blastp blastall -p blastp -d "swiss" -i p123456.seq sge-blast-with-jobname.sh qstat job-id prior name user state submit/start at queue --------------------------------------------------------------------------------- 3754228 0.00000 blastp userid qw 02/01/2012 12:13:22 83, HPC in Life Sciences

SGE example redirect output/error Write stdout and stderr in specific files #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd #$ -o blast-output.txt #$ -e blast-error.txt standard'output(is'redirected' to'a'file'indicated'by'yo' standard'error'indicated'by'ye' blastall -p blastp -d "swiss" -i p123456.seq sge-blast-redirect.sh ls blast-output.txt blast-error.txt 84, HPC in Life Sciences

SGE example use job ID in output file name Distinguish jobs and files #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd #$ -o $JOB_ID-output.txt #$ -e $JOB_ID-error.txt blastall -p blastp -d "swiss" -i p123456.seq echo "My JobID was $JOB_ID" sge-blast-with-jobid.sh cat 3754473-output.txt My JobID was 3754473 85, HPC in Life Sciences

SGE example e-mail notification Receive an e-mail when job has finished Includes CPU usage information (timing) etc. #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd #$ -M MyEmal@example.org #$ -m e b' 'beginning'of'job' e' 'end' a' 'aborted' s' 'suspended' blastall -p blastp -d "swiss" -i p123456.seq sge-blast-with-email.sh See example e-mail on next slide 86, HPC in Life Sciences

eamail$is$sent$by$sge$server$$ (root,$gridamonitor@id.unibe.ch)$ Job 3755460 (blast-with-email.sh) Complete User = userid Queue = short.q@cnode03.ubelix.unibe.ch Host = cnode03.ubelix.unibe.ch Start Time = 02/01/2012 14:22:33 End Time = 02/01/2012 14:22:35 User Time = 00:00:01 System Time = 00:00:00 Wallclock Time = 00:00:02 CPU = 00:00:02 Max vmem = 320.426M Exit Status = 0 87, HPC in Life Sciences

SGE Array Job: submit the same job multiple times Motivation/example: My simulation software mysim needs to be executed 10 times Submit a single job rather than 10 individual ones #!/bin/bash #$ -t 1-10 #$ -l h_cpu=24:00:00 #$ -l h_vmem=100m #$ -cwd #$ -N array-job Index/Task$in'job'array' echo "Task $SGE_TASK_ID out of 10 mysim --seed $SGE_TASK_ID sge-array-job.sh 88, HPC in Life Sciences

Job arrays (cont.) qstat job-id prior name user state submit/start at queue slots ja-task-id --------------------------------------------------------------------------------- 375 0.94978 array-job userid r 02/01/2012 15:59:18 all.q@cnode45 1 3 375 0.48058 array-job userid t 02/01/2012 15:59:18 all.q@cnode45 1 4 375 0.32416 array-job userid r 02/01/2012 15:59:18 all.q@dnode17 1 5 375 0.24594 array-job userid qw 02/01/2012 15:58:49 1 6-10:1 Same job ID for all tasks! 89, HPC in Life Sciences

Resource requirements Job can explicitly require certain hardware resources such as RAM (main memory), swap space, etc. #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd echo "hostname" hostname xk'(kb),'xm'(mb)','xg'(gb)' ' 90, HPC in Life Sciences

Resource requirements: local disk space Usage of scratch space $TMP #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -l scratch=1 #$ -l scratch_size=1g #$ -l scratch_files=1 mkdir $TMP/hallo cd $TMP/hallo/ pwd ls -l $TMP/hallo xk'(kb),'xm'(mb),'xg'(gb)' sge-scratch-space.sh cat scratch-space.sh.o3756586 /scratch/local/3756586.1.all.q/hallo total 0 91, HPC in Life Sciences

Multi-threaded jobs (1 thread per core) By default, SGE reserves 1 core (CPU) per job Multi-threaded jobs need to request more cores (slots) #!/bin/bash #$ -l h_cpu=01:00:00 #$ -l h_vmem=1g #$ -cwd #$ -pe smp 4./pthread-4 4'CPU'cores'(processors)'requested' ' pe=programming'environment' smp=symmetric'mul:'processing' ' sge-4-threads.sh job-id prior name user state submit/start at queue slots --------------------------------------------------------------------------------------------- 3772866 0.85538 sge- userid r 02/03/2012 10:06:33 mpi.q@enode13.ubelix.unibe.ch 4 92, HPC in Life Sciences

Fribourg Bern: additional cluster in Bern Managed by Interfaculty Bioinformatics unit http://www.biology.unibe.ch/content/bioinformatics Contact: Remy.Brugmann@biology.unibe.ch 8 machines with 240 cores (SGE) 256 512 GB RAM per machine ssh username@binfservms01.unibe.ch 93, HPC in Life Sciences

Bern: additional cluster in Bern (2) /home/username For scripts and other small files (quota 20 GB) Will be backed up soon (daily?) /data3/users/<username> Location to store your data Quota 5TB; not backed up! /scratch (local to node, local disk) (4.8TB on 80 core nodes, 4TB on 16 core nodes) 94, HPC in Life Sciences

Bern: additional cluster in Bern (3) SGE example #!/bin/sh #$ -S /bin/sh #$ -e scriptname-$job_id.err #$ -o scriptname-$job_id.out #$ -M remy.bruggmann@biology.unibe.ch #$ -l h_vmem=50g #$ -l h_rt=540:00:00 #$ -pe smp 8 #$ -q all.q # here starts the command that is executed on the cluster blastall p blastn -a 8 -i seqs.fasta -d blastdatabase -o blastout 95, HPC in Life Sciences

Basic overview /home/<userid> private scripts, small data /scratch/ programs should write results there /archive/... long term storage /db/ public bioinformatics databases http://www.vital-it.ch/support/storage_guidelines.php 97, HPC in Life Sciences

Home directories /home/<userid> Backed up on tape 5 GB quota Only most important files should be stored there Your files we don t support software that you install there! Please do not store large (temporary) results in /home but do so in the /scratch directory (next slide) Check quota with quota s Filesystem blocks quota limit grace files quota limit grace 10.0.0.41:/exports/home 828M 5120M 5632M 2009 0 0 Space'used'' 5'GB'quota' Number'of'files' created' 98, HPC in Life Sciences

Space Limitation for LSF: #!/bin/bash #BSUB -L /bin/bash #BSUB -o blast-output.txt By default, output/error redirection is written into the following directory (note quota on $HOME): $HOME/.lsbatch/ If output is large (>5GB), that will exceed your quota in your home directory so better write to a file in /scratch without using o myprogram > /scratch/cluster/daily/userid/output.txt (-o is only written, once job has finished) 99, HPC in Life Sciences

Scratch directories /scratch No backup! automatically cleaned! 1 TB quota per user (/scratch/cluster) currently 56 TB /scratch/cluster/[daily weekly monthly] Create a subdirectory according to your userid You can work like on a single machine but make sure that different processes don t write into the same file! Once you have finished your computation, please download results from /scratch to your local machine and back up your results /scratch/local/[daily weekly] only 20-50 GB Local scratch on compute node (not shared!) Very fast access! 100, HPC in Life Sciences

LSF Job to use /scratch/ Explicitly cd into /scratch or write files directly there #!/bin/bash #BSUB -L /bin/bash #BSUB -o blast-output.txt #BSUB -e blast-error.txt #BSUB -J myjob Could'also'be'redirected' to'/scratch.'by'default,' local'directory'from'where' you'submiqed'job!' cd /scratch/cluster/daily/userid blastall -p blastp -d "swiss" -i p123456.seq Change'working' directory'to'write' results'directly' into'''/scratch' 101, HPC in Life Sciences

Attention: automatic cleanup in /scratch/cluster/[daily weekly monthly]/ Often, when you retrieve external data in compressed format (e.g. tar.gz) the files are "older" than 1 week, 1 month etc. tar -zxvf mydata.tar.gz (from e.g. from 2012) will preserve file creation time you will loose the data next day! Solution: tar zxvmf mydata.tar.gz Use 'm' option to extract file setting current time (Note: here, the order of the options is important) 102, HPC in Life Sciences

Archive /archive Large storage for scientific data (more than ½ PB) Automatic backup and versioning Hierarchical storage system (HSM) with disk and tape No quota but if disk cache is full, files are truncated from disk and must be retrieved from tape Don t run programs that write directly to archive! Copy data explicitly from/to archive (zip/tarball data!) Avoid copying/archiving many small files, i.e. create larger files! /archive/<entity>/<group>/<userid> $HOME/archive (symbolic link in your home directory) 103, HPC in Life Sciences

Hierarchical Storage System (HSM) Quantum StorNext basics: Clients HSM Clients HSM MDC * SMB/CIFS NFS SAN Network Disk Storage Tape Storage * MetaData Controller Slide by Volker Flegel 104, HPC in Life Sciences

Is your file in disk cache of HSM? ondisk Check, if a file is on the disk cache of the HSM or truncted (max 16 KB on disk) ondisk -h ondisk../authpaper.pdf: on_disk./bigresults.pdf: truncated./ieeetrans.bst: truncated./ocg.sty: on_disk./literatur.bib: on_disk./bigresults.tex: truncated 105, HPC in Life Sciences

Archive: ask LSF to transfer/stage files Before jobs starts, LSF can copy the file to a scratch directory #/bin/bash source'file'on'/archive' #BSUB -L /bin/bash #BSUB -o output.txt #BUSB -e error.txt #BSUB -J stage3 #BSUB -f "/archive/ /file1.seq > /scratch/cluster/ /file1.seq" #BSUB -f "/archive/ /file2.seq > /scratch/cluster/ /file2.seq" #BSUB -f "/archive/ /file3.seq > /scratch/cluster/ /file3.seq" cd /scratch/cluster/ / ls -l tail file1.seq des:na:on'file'on/scratch' 106, HPC in Life Sciences

Data access latency: important change! Hierarchical storage system: Provides access to large amounts of data (/archive) Data might not be always available on disk but need to be fetched from a tape drive access latency storage capacity (increase) RAM disk cache tape (archive) storage cost (increase) Keep'that'hierarchy'in'mind'when'you'plan'to'use'data'in'archive' directory!!!'' 107, HPC in Life Sciences

Summary Vital-IT (UNIL/EPFL/UNIGE) and UBELIX Task$ LSF$ SGE$ Submit'job' bsub' qsub' Check'status' bjobs' qstat' Cancel/Kill'job' bkill' qdel' VitalYIT'soWware'on'all'clusters' 3 clusters at 3 sites Medium and large size memory Disk storage + long term tape archive Single cluster Medium size memory Disk space only 108, HPC in Life Sciences

Number and size of files (per directory) Note that file systems are not good at dealing with thousands of thousands of small files Particularly true for cluster/parallel file systems! Advice: try to organise your data to keep larger files (bigger than 4 MB!) If you store files, create tar balls for smaller files to have less than ~10 000 files per directory 109, HPC in Life Sciences

Conclusion Vital-IT software at 4 sites: UNIL, EPFL, UNIGE and UNIBE Infrastructure summary Submit jobs from front-end nodes Be aware of quotas and different access latencies /scratch for writing data (UNIL/EPFL/UNIGE) Contact us for help etc. Vital-IT is much more than an HPC infrastructure We work closely with biologist and platforms Common research projects, embedded bioinformaticians 110, HPC in Life Sciences

Thank you 111, HPC in Life Sciences