Introduction to High-Performance Computing and the Supercomputing Institute Carlos P Sosa Cray, Inc. and Biomedical Informatics and Computational Biology University of Minnesota Rochester
Agenda What is High-Performance Computing? Floating-Point Operations Cost per Gflops High-Performance Computing Historical Perspective Introduction to MSI facilities Show you how to access our systems Point you to where to go for help Brief introduction to Linux/UNIX and some useful commands Ensure that you are not overwhelmed Encourage you to ask questions of MSI staff to get what you need 2
What is High-Performance Computing? High-performance computing (HPC) uses supercomputers to solve advanced computation problems Today, computer systems approaching the teraflops-region are counted as HPC-computers Measure of a computer's processor speed. This speed can be expressed as a trillion floating point operations per second, 1012 floating-point operations per second http://en.wikipedia.org/wiki/high-performance_computing 3
International System of Units April 13 Cray Inc. 4
Floating-Point Operations FLOPS (or flops or flop/s, for floating-point operations per second) Measure of a computer's performance Floating-point is a method of representing real numbers S stands for "second", conservative speakers consider "FLOPS" as both the singular and plural of the term FLOP (or flop) is used as an abbreviation for "FLoating-point OPeration Flop count is a count of these operations in a given section of a computer program FLOPS is not an SI unit http://en.wikipedia.org/wiki/flops 5
Why Floating Point significant digits x base exponent radix point (decimal point, or, more commonly in computers, binary point) can "float or be shifted left or right http://en.wikipedia.org/wiki/floating-point 6
How to Calculate Mflops? 10 7
What s a petaflop? One quadrillion calculations per second! If you multiplied two 14-digit numbers together per second: 32 years to complete 1 billion calculations. 32 thousand years to complete 1 trillion calculations. 32 million years to complete 1 quadrillion calculations. 32 years ago, Star Wars was released 32 thousand years ago, early cave paintings were completed 32 million years ago, the Alps were rising in Europe Courtesy of NCSA: Blue Waters Project
Binary versus Decimal 2 10 is very nearly equal to 1000 and started using the SI prefix "kilo" to mean 1024 Everybody does not "know" what a megabyte is Computer memory, most manufacturers use megabyte to mean 2 20 = 1 048 576 bytes Manufacturers of computer storage devices usually use the term to mean 1 000 000 bytes Local area networks have used megabit per second to mean 1 048 576 bit/s Telecommunications engineers use it to mean 10 6 bit/s Two definitions of the megabyte are not enough, a third megabyte of 1 024 000 bytes is the megabyte used to format the familiar 90 mm (3 1/2 inch), "1.44 MB" diskette The confusion is real, as is the potential for incompatibility in standards and in implemented systems. http://physics.nist.gov/cuu/units/binary.html 9
Binary Notation April 13 http://physics.nist.gov/cuu/units/binary.html Cray Inc. 10
Historical Change in Cost per GFlops http://en.wikipedia.org/wiki/flops April 13 Cray Inc. 11
HPC Historical Perspective High-Performance Computers were introduced in the 1960s and were designed primarily by Seymour Cray at Control Data Corporation (CDC) Led the market into the 1970s Founded Cray Research Big irons dominated the market (1985 1990) 1980s the decade of the minicomputer Mid-1990s "supercomputer market crash" 12
Big Irons Source: http://www.mrynet.com/cray/docs.html 1985 Cray-2 April 13 Cray Inc. 13
Supercomputers Peak Performance April 13 http://www.reed-electronics.com/electronicnews/article/ca508575.html?indust ryid=21365 From Kilobytes to Petabytes in 50 Years: http://www.eurekalert.org/features/doe/2002-03/dlnl-fkt062102.php Cray Inc. 14
Why Should I Care About HPC? April 13, 2009 The Looming Petascale Chemists gear up for a new generation of supercomputers The new petascale computers will be 1,000 times faster than the terascale supercomputers of today, performing more than 1,000 trillion operations per second. And instead of machines with thousands of processors, petascale machines will have many hundreds of thousands that simultaneously process streams of information. http://pubs.acs.org/cen/science/87/8715sci3.html This technological sprint could be a great boon for chemists, allowing them to computationally explore the structure and behavior of bigger and more complex molecules. April 13 Cray Inc. 15
Towards More Realistic Systems April 13 Enabling and Scaling Biomolecular Simulations of 100 Million Atoms on Petascale Machines with a Multicore-optimized Message-driven Runtime, Chao Mei, Yanhua Sun, Gengbin Zheng, Eric J. Bohm, Laxmikant V. Kale, James C.Phillips, Chris Harrison, SC 11 November 12-18, 2011, Seattle, Washington Cray Inc. 16
Introduction to the Minnesota Supercomputing Institute (MSI)
MSI At a Glance Manages approximately 400 software packages and 50 research databases. Main data center in Walter Library. MSI operates five laboratories on campus, mostly serving the Life Sciences. Serves more than 500 PI groups with 3000+ users. MSI is an academic unit of the University of Minnesota (under OVPR) with about 45 full time employees. Dedicated resources for serial workflows, database management, and cloud access.
BSCL MSI Offices HPC Resources Labs
Eligibility Faculty members at the University of Minnesota University of Minnesota academic professionals Faculty researchers at other accredited institutions of post-secondary education in the state of Minnesota Some software may not be available to users not affiliated with UMN
MSI Resources HPC Resources Koronis Itasca Calhoun Cascade GPUT Laboratories BMSDL BSCL CGL SDVL LMVL Software Chemical and Physical Sciences Engineering Graphics and Visualization Life Sciences Development Tools User Services Consulting Tutorials Code Porting Parallelization Visualization
Access System access information is available at www.msi.umn.edu/access.html mymsi for additional resources For building access, contact help@msi.umn.edu x6-0802 Information can be found in the General Information page for each lab (www.msi.umn.edu/labs/ ) For help, contact help@msi.umn.edu x6-0802
Remote Access to MSI lab Linux systems Cannot remotely login directly to most machines at MSI Two options to connect to MSI lab systems ssh login.msi.umn.edu NX - Requires user to download and install client Cannot run software directly from login or NX isub www.msi.umn.edu/remote-access
HPC Resources Koronis: SGI Altix 1140 Intel Nehalem Cores 2.96 TB of memory Itasca: Hewlett-Packard 3000BL 8728 Intel Nehalem Cores 26 TB of memory Calhoun: SGI Altix XE 1300 1440 Intel Xeon Clovertown Cores 2.8 TB of memory Cascade: 8 Dell Compute Nodes 32 Nvidia M2070 GPGPUs GPUT: 16 Nvidia GeForce GTX 480 GPUs
Laboratories Scientific Development and Visualization Application development Computational Physics Workshops Basic Sciences Computing Structural Biology Batch processing Stereo Projection system Computational Genetics Bioinformatics Genomics Proteomics Biomedical Modeling, Simulation, and Design Drug Design Molecular Modeling Computational chemistry LCSE-MSI Visualization Virtual reality Large screen Remote visualization
SDVL LMVL BSCL BSCL
BMSDL
St. Paul Campus CGL
Software Approximately 400 software applications www.msi.umn.edu/sw Classified into three service levels www.msi.umn.edu//resources/software/service-levels Primary Ancillary Minimal Purchase justification required
Disk Resources Labs and HPC systems (other than Koronis) have shared home directories Disk space is allocated to a group 50 GB initial quota on Koronis. Every lab and HPC machine has access to at least 1 TB of scratch space On HPC systems your scratch quota is 50 GB You may request a quota increase if necessary. PIs may request project space for large data and group collaboration. www.msi.umn.edu/resources/storage/
Tutorials/Workshops Introductory Unix, Linux, remote computing, job submission, queue policy Programming & Scientific Computation Code parallelization, programming languages, math libraries Computational Physics Fluid dynamics, space physics, structural mechanics, material science Computational Chemistry Quantum chemistry, classical molecular modeling, drug design, cheminformatics Computational Biology Structural biology, computational genomics, proteomics, bioinformatics www.msi.umn.edu/tutorial
Support MSI is a resource dedicated to enabling and improving the quality of computational research at the University of Minnesota Basic research support is made available to researchers at minimal or no cost MSI participates with researchers who are seeking funding for projects that require MSI consultants or developers
Projects Internal Services Assist with long term development projects Wide range of project activities Tailored to specific research initiatives or development programs At cost External Services www.msi.umn.edu/services Acknowledgement of MSI in publications / grant proposals is requested.
Integrate MSI Resources into Course Work Class accounts Access to MSI on a semester basis for all students in a course Customized workshops and guest lectures When our expertise overlaps with your interests, e.g., parallel programming, integration of computational assignments in a field Reserve the LMVL Use the stereoscopic capabilities of the visualization lab for high impact structure and model visualization lectures
MSI web pages MSI home page www.msi.umn.edu Software www.msi.umn.edu/sw Password reset www.msi.umn.edu/password Tutorials www.msi.umn.edu/tutorial FAQ www.msi.umn.edu/support/faq.html
Why Modules? Modules environment management package provides support for dynamic modification of the user environment via modulefiles Each modulefile contains all the information needed to configure the shell for a particular application Advantage: You are not required to specify explicit paths for different executable versions or to set the $MANPATH and other environment variables manually. All the information is embedded in the modulefile Modules can be loaded and unloaded dynamically and atomically, in an clean fashion. All popular shells are supported, including bash, ksh, zsh, sh, csh, tcsh, as well as some scripting languages such as perl As a user, you can add and remove modulefiles from your current shell environment The environment changes performed by a modulefile can be viewed by using the module command as well 36
Module Commands Command module list module avail module avail -l module display module load module unload module help module swap Description Lists modules currently loaded in a user s environment Lists all available modules on a system in condensed format Lists all available modules on a system in long format Shows environment changes that will be made by loading a given module Loads a module Unloads a module Shows help for a module Swaps a currently loaded module for an unloaded module 37
What is Loaded Now? cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct 38
Re-Initializing the Module Command Modules software functionality is highly dependent upon the shell environment being used Sometimes when switching between shells, modules must be reinitialized For example, you might see an error such as the following: $ module list -bash: module: command not found To fix this, just re-initialize your modules environment: $ source $MODULESHOME/init/myshell Where myshell is the name of the shell you are using and need to re-initialize 39
What is Available? To see which modulefiles are available on your system, enter this command: % module avail [string] The module avail command produces an alphabetical listing of every modulefile in your module use path and has no option for "grepping." Therefore, it is usually more useful to use the command with an string argument cpsosa@silver:~> module avail gcc ------/soft/modules/modulefiles ------- gcc/4.3.3 gcc/4.4.0 gcc/4.4.3(default) 40
Loading and Unloading Modules If a modulefile is not already loaded, use the module load command to load it. % module load modulefile This command loads the currently defined default version of the module, unless you specify otherwise cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct cpsosa@silver:~> module load dock/6.3 cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 4) moab 7) base 10) ompi/1.2.9/xl 2) vars 5) torque 8) xlc/10.1 11) dock/6.3 3) user 6) suacct 9) xlf/12.1 cpsosa@silver:~> module unload dock/6.3 41
Module Swapping Alternatively, you can use the module swap or module switch command to unload one module and load the comparable module cpsosa@silver:~> module load gcc/4.3.3 cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct 8) gcc/4.3.3 cpsosa@silver:~> module swap gcc/4.4.0 cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct 8) gcc/4.4.0 42
Lab 1: Test Modules 1.$ssh itasca 2.$module list 3. Type: $g09 4. Type: module load g09 5. Type: g09 43
Portable Batch System (PBS) https://www.msi.umn.edu/labs/pbs#intro The Portable Batch System (PBS) is a queuing system installed for job batch processing It matches job requirements with available resources It ensures that machines are fully used and resources are distributed among all users In contrast to the HPC system queues, the Lab system queues do not require Service Units (SUs) in order for jobs to run http://wiki.ccs.tulane.edu/index.php5/portable_batch_system_%28pbs%29 44
Lab Compute Nodes Node Details Nodes Model CPU cores Memory labh01 labh02 labh03 - labh08 PowerEdge R900 SunFire X4440 SunFire X4600 24 Intel Xeon 2.67 GHz 16 AMD Opteron 8384 2.7 GHz 32 AMD Opteron 8356 2.3GHz 128GB 128GB 128GB mirror1-mirror16 Altix XE 310 8 Intel Xeon 2.66GHz 16GB lab001-lab064 Altix XE 310 8 Intel Xeon 2.66GHz 16GB laboc01-laboc04 LiquidCool Liquid Submerged 12 Intel Xeon X5690 @ 4.1GHz 24GB 45
How to Create a Job Submission Script 1. Log in to lab.msi.umn.edu 2. Write a PBS batch script 3. Example: 1. 1-hour job to run on a single processor of a single node 2. 1gb of memory 3. Load any preferred modules 4. run any software that can operate in batch mode #!/bin/bash -l #PBS -l nodes=1:ppn=1,mem=1gb,walltime=01:00:00 #PBS -m abe cd /home/msi/username/testpbs module load intel./test < input.dat > output.dat 46
How to Submit a Job Use the command qsub to submit a job to the queuing system qsub takes a job submission script that contains special commands telling PBS what resources are needed It also contains the commands necessary to run the submitted job. Example: $qsub script.pbs To submit to a different queue: $qsub -q lab-long script.pbs 47
How to Check Job Status You can check job status using the showq command $showq -u username 48
Lab1: submit a very simple script Go to the scratch file system: $cd /scratch Create a directory: $mkdir myaccount Move to that directory: $cd myaccount Create a directory:$mkdir test Move to that directory: $cd test Copy file: $Cp ~cpsosa/gaussian/test/*. Submit your first script: $Qsub test.pbs 49
Example: Electronic Structure Calculation http://www.chem.wayne.edu/~hbs/chm6440/pes.html 50
Schrödinger Equation http://www.cobalt.chem.ucalgary.ca/ziegler/wilson/qm/61.schrodingerequation.jpeg N-body wavefunction Energy of either the ground or an excited state of the system charges of the individual particles Spatial positions http://www.physics.uc.edu/~pkent/thesis/pkthnode12.html 51
Gaussian09 an Electronic Structure Suite of Programs Gaussian 09: is an electronic structure package capable of predicting many properties of atoms, molecules, and reactions, e.g. molecular energies and structures vibrational frequencies molecular orbitals and much more utilizing ab initio, density functional theory, semiempirical, molecular mechanics, and hybrid methods. 52
Gaussian History Gaussian70, Gaussian76, Gaussian77, Gaussian78, Gaussian80, Gaussian82, Gaussian83, Gaussian85, Gaussian86, Gaussian88, Gaussian90, Gaussian 92, Gaussian93, Gaussian 94, Gaussian95, Gaussian96, Gaussian 98, Gaussian 03, Gaussian 09 Born Died Nationality Fields Alma mater Doctoral advisor John Pople 31 October 1925 Burnham-on-Sea, Somerset, England March 15, 2004 (aged 78) Chicago, Illinois, United States England Theoretical chemistry Quantum chemistry Computational chemistry Cambridge University John Lennard-Jones Notable awards Nobel Prize in Chemistry (1998) 53
Input Format % Resource management # Route card Title section Molecular coordinates Geometric variables Other input options blank line blank line blank line blank line blank line 54
Input File %mem=32mb #p hf/6-31g opt hf/6-31g optimization of water 0 1 o h 1 oh h 1 oh 2 aoh oh=0.9 aoh=104.0 system resources computational model type of calculation title charge & multiplicity structure definition (z-matrix) variable values 55
Route Card Description: specifies keywords and options always begins with a # character keywords can be specified in any order options are grouped in parentheses, () keywords should not be repeated route section can be up to 5 lines long ended with a blank line Syntax: #[p] method/basis [keyword=(opt1,opt2)] [keyword2=(opt1[,opt2])] 56
Type of Calculation single point energy and properties geometry optimization reaction path following/searching frequency 57
Computational Method Level of theory: molecular mechanics amber, dreiding, uff semi-empirical am1, pm3, mndo, density functional theory b3lyp, mpwpw91, custom ab initio hf, mp2, ccsd, qcisd, hybrid g2, g3, oniom, 58
Lab3: Run Gaussian09 1. Copy files: $cp ~cpsosa/gaussian/g09/*. 2. Build your h2o.com input file 3. Build your h2o.pbs 4. Submit your h2o.pbs to the QUE 5. Visualize results 59
Lab 4: Visualize Results 1. Load the gaussian module if you do not have already loaded: $module load gaussian 2. Run GaussView by typing: $gview 60
Questions? MSI help desk is staffed Monday through Friday from 8:30AM to 7:00PM Walk-in help available in room 569 Walter Phone 612.626.0802 Email help@msi.umn.edu
http://r.umn.edu/academics-research/bicb/ Thank You! Many thanks to: Nancy Rowe for providing The MSI slides MSI Support http://www.msi.umn.edu http://www.cray.com 62