Introduction to High-Performance Computing and the Supercomputing Institute

Transcription

1 Introduction to High-Performance Computing and the Supercomputing Institute Carlos P Sosa Cray, Inc. and Biomedical Informatics and Computational Biology University of Minnesota Rochester

2 Agenda What is High-Performance Computing? Floating-Point Operations Cost per Gflops High-Performance Computing Historical Perspective Introduction to MSI facilities Show you how to access our systems Point you to where to go for help Brief introduction to Linux/UNIX and some useful commands Ensure that you are not overwhelmed Encourage you to ask questions of MSI staff to get what you need 2

3 What is High-Performance Computing? High-performance computing (HPC) uses supercomputers to solve advanced computation problems Today, computer systems approaching the teraflops-region are counted as HPC-computers Measure of a computer's processor speed. This speed can be expressed as a trillion floating point operations per second, 1012 floating-point operations per second 3

4 International System of Units April 13 Cray Inc. 4

5 Floating-Point Operations FLOPS (or flops or flop/s, for floating-point operations per second) Measure of a computer's performance Floating-point is a method of representing real numbers S stands for "second", conservative speakers consider "FLOPS" as both the singular and plural of the term FLOP (or flop) is used as an abbreviation for "FLoating-point OPeration Flop count is a count of these operations in a given section of a computer program FLOPS is not an SI unit 5

6 Why Floating Point significant digits x base exponent radix point (decimal point, or, more commonly in computers, binary point) can "float or be shifted left or right 6

7 How to Calculate Mflops? 10 7

8 What s a petaflop? One quadrillion calculations per second! If you multiplied two 14-digit numbers together per second: 32 years to complete 1 billion calculations. 32 thousand years to complete 1 trillion calculations. 32 million years to complete 1 quadrillion calculations. 32 years ago, Star Wars was released 32 thousand years ago, early cave paintings were completed 32 million years ago, the Alps were rising in Europe Courtesy of NCSA: Blue Waters Project

9 Binary versus Decimal 2 10 is very nearly equal to 1000 and started using the SI prefix "kilo" to mean 1024 Everybody does not "know" what a megabyte is Computer memory, most manufacturers use megabyte to mean 2 20 = bytes Manufacturers of computer storage devices usually use the term to mean bytes Local area networks have used megabit per second to mean bit/s Telecommunications engineers use it to mean 10 6 bit/s Two definitions of the megabyte are not enough, a third megabyte of bytes is the megabyte used to format the familiar 90 mm (3 1/2 inch), "1.44 MB" diskette The confusion is real, as is the potential for incompatibility in standards and in implemented systems. 9

10 Binary Notation April 13 Cray Inc. 10

11 Historical Change in Cost per GFlops April 13 Cray Inc. 11

12 HPC Historical Perspective High-Performance Computers were introduced in the 1960s and were designed primarily by Seymour Cray at Control Data Corporation (CDC) Led the market into the 1970s Founded Cray Research Big irons dominated the market ( ) 1980s the decade of the minicomputer Mid-1990s "supercomputer market crash" 12

13 Big Irons Source: Cray-2 April 13 Cray Inc. 13

14 Supercomputers Peak Performance April 13 ryid=21365 From Kilobytes to Petabytes in 50 Years: Cray Inc. 14

15 Why Should I Care About HPC? April 13, 2009 The Looming Petascale Chemists gear up for a new generation of supercomputers The new petascale computers will be 1,000 times faster than the terascale supercomputers of today, performing more than 1,000 trillion operations per second. And instead of machines with thousands of processors, petascale machines will have many hundreds of thousands that simultaneously process streams of information. This technological sprint could be a great boon for chemists, allowing them to computationally explore the structure and behavior of bigger and more complex molecules. April 13 Cray Inc. 15

16 Towards More Realistic Systems April 13 Enabling and Scaling Biomolecular Simulations of 100 Million Atoms on Petascale Machines with a Multicore-optimized Message-driven Runtime, Chao Mei, Yanhua Sun, Gengbin Zheng, Eric J. Bohm, Laxmikant V. Kale, James C.Phillips, Chris Harrison, SC 11 November 12-18, 2011, Seattle, Washington Cray Inc. 16

17 Introduction to the Minnesota Supercomputing Institute (MSI)

18 MSI At a Glance Manages approximately 400 software packages and 50 research databases. Main data center in Walter Library. MSI operates five laboratories on campus, mostly serving the Life Sciences. Serves more than 500 PI groups with users. MSI is an academic unit of the University of Minnesota (under OVPR) with about 45 full time employees. Dedicated resources for serial workflows, database management, and cloud access.

19 BSCL MSI Offices HPC Resources Labs

20 Eligibility Faculty members at the University of Minnesota University of Minnesota academic professionals Faculty researchers at other accredited institutions of post-secondary education in the state of Minnesota Some software may not be available to users not affiliated with UMN

21 MSI Resources HPC Resources Koronis Itasca Calhoun Cascade GPUT Laboratories BMSDL BSCL CGL SDVL LMVL Software Chemical and Physical Sciences Engineering Graphics and Visualization Life Sciences Development Tools User Services Consulting Tutorials Code Porting Parallelization Visualization

22 Access System access information is available at mymsi for additional resources For building access, contact x Information can be found in the General Information page for each lab ( ) For help, contact [email protected] x6-0802

23 Remote Access to MSI lab Linux systems Cannot remotely login directly to most machines at MSI Two options to connect to MSI lab systems ssh login.msi.umn.edu NX - Requires user to download and install client Cannot run software directly from login or NX isub

24 HPC Resources Koronis: SGI Altix 1140 Intel Nehalem Cores 2.96 TB of memory Itasca: Hewlett-Packard 3000BL 8728 Intel Nehalem Cores 26 TB of memory Calhoun: SGI Altix XE Intel Xeon Clovertown Cores 2.8 TB of memory Cascade: 8 Dell Compute Nodes 32 Nvidia M2070 GPGPUs GPUT: 16 Nvidia GeForce GTX 480 GPUs

25 Laboratories Scientific Development and Visualization Application development Computational Physics Workshops Basic Sciences Computing Structural Biology Batch processing Stereo Projection system Computational Genetics Bioinformatics Genomics Proteomics Biomedical Modeling, Simulation, and Design Drug Design Molecular Modeling Computational chemistry LCSE-MSI Visualization Virtual reality Large screen Remote visualization

26 SDVL LMVL BSCL BSCL

27 BMSDL

28 St. Paul Campus CGL

29 Software Approximately 400 software applications Classified into three service levels Primary Ancillary Minimal Purchase justification required

30 Disk Resources Labs and HPC systems (other than Koronis) have shared home directories Disk space is allocated to a group 50 GB initial quota on Koronis. Every lab and HPC machine has access to at least 1 TB of scratch space On HPC systems your scratch quota is 50 GB You may request a quota increase if necessary. PIs may request project space for large data and group collaboration.

31 Tutorials/Workshops Introductory Unix, Linux, remote computing, job submission, queue policy Programming & Scientific Computation Code parallelization, programming languages, math libraries Computational Physics Fluid dynamics, space physics, structural mechanics, material science Computational Chemistry Quantum chemistry, classical molecular modeling, drug design, cheminformatics Computational Biology Structural biology, computational genomics, proteomics, bioinformatics

32 Support MSI is a resource dedicated to enabling and improving the quality of computational research at the University of Minnesota Basic research support is made available to researchers at minimal or no cost MSI participates with researchers who are seeking funding for projects that require MSI consultants or developers

33 Projects Internal Services Assist with long term development projects Wide range of project activities Tailored to specific research initiatives or development programs At cost External Services Acknowledgement of MSI in publications / grant proposals is requested.

34 Integrate MSI Resources into Course Work Class accounts Access to MSI on a semester basis for all students in a course Customized workshops and guest lectures When our expertise overlaps with your interests, e.g., parallel programming, integration of computational assignments in a field Reserve the LMVL Use the stereoscopic capabilities of the visualization lab for high impact structure and model visualization lectures

35 MSI web pages MSI home page Software Password reset Tutorials FAQ

36 Why Modules? Modules environment management package provides support for dynamic modification of the user environment via modulefiles Each modulefile contains all the information needed to configure the shell for a particular application Advantage: You are not required to specify explicit paths for different executable versions or to set the $MANPATH and other environment variables manually. All the information is embedded in the modulefile Modules can be loaded and unloaded dynamically and atomically, in an clean fashion. All popular shells are supported, including bash, ksh, zsh, sh, csh, tcsh, as well as some scripting languages such as perl As a user, you can add and remove modulefiles from your current shell environment The environment changes performed by a modulefile can be viewed by using the module command as well 36

37 Module Commands Command module list module avail module avail -l module display module load module unload module help module swap Description Lists modules currently loaded in a user s environment Lists all available modules on a system in condensed format Lists all available modules on a system in long format Shows environment changes that will be made by loading a given module Loads a module Unloads a module Shows help for a module Swaps a currently loaded module for an unloaded module 37

38 What is Loaded Now? module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct 38

39 Re-Initializing the Module Command Modules software functionality is highly dependent upon the shell environment being used Sometimes when switching between shells, modules must be reinitialized For example, you might see an error such as the following: $ module list -bash: module: command not found To fix this, just re-initialize your modules environment: $ source $MODULESHOME/init/myshell Where myshell is the name of the shell you are using and need to re-initialize 39

40 What is Available? To see which modulefiles are available on your system, enter this command: % module avail [string] The module avail command produces an alphabetical listing of every modulefile in your module use path and has no option for "grepping." Therefore, it is usually more useful to use the command with an string argument cpsosa@silver:~> module avail gcc /soft/modules/modulefiles gcc/4.3.3 gcc/4.4.0 gcc/4.4.3(default) 40

41 Loading and Unloading Modules If a modulefile is not already loaded, use the module load command to load it. % module load modulefile This command loads the currently defined default version of the module, unless you specify otherwise cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct cpsosa@silver:~> module load dock/6.3 cpsosa@silver:~> module list Currently Loaded Modulefiles: 1) local 4) moab 7) base 10) ompi/1.2.9/xl 2) vars 5) torque 8) xlc/ ) dock/6.3 3) user 6) suacct 9) xlf/12.1 cpsosa@silver:~> module unload dock/6.3 41

42 Module Swapping Alternatively, you can use the module swap or module switch command to unload one module and load the comparable module module load gcc/4.3.3 module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct 8) gcc/4.3.3 module swap gcc/4.4.0 module list Currently Loaded Modulefiles: 1) local 3) user 5) torque 7) base 2) vars 4) moab 6) suacct 8) gcc/

43 Lab 1: Test Modules 1.$ssh itasca 2.$module list 3. Type: $g09 4. Type: module load g09 5. Type: g09 43

44 Portable Batch System (PBS) The Portable Batch System (PBS) is a queuing system installed for job batch processing It matches job requirements with available resources It ensures that machines are fully used and resources are distributed among all users In contrast to the HPC system queues, the Lab system queues do not require Service Units (SUs) in order for jobs to run 44

45 Lab Compute Nodes Node Details Nodes Model CPU cores Memory labh01 labh02 labh03 - labh08 PowerEdge R900 SunFire X4440 SunFire X Intel Xeon 2.67 GHz 16 AMD Opteron GHz 32 AMD Opteron GHz 128GB 128GB 128GB mirror1-mirror16 Altix XE Intel Xeon 2.66GHz 16GB lab001-lab064 Altix XE Intel Xeon 2.66GHz 16GB laboc01-laboc04 LiquidCool Liquid Submerged 12 Intel Xeon 4.1GHz 24GB 45

46 How to Create a Job Submission Script 1. Log in to lab.msi.umn.edu 2. Write a PBS batch script 3. Example: 1. 1-hour job to run on a single processor of a single node 2. 1gb of memory 3. Load any preferred modules 4. run any software that can operate in batch mode #!/bin/bash -l #PBS -l nodes=1:ppn=1,mem=1gb,walltime=01:00:00 #PBS -m abe cd /home/msi/username/testpbs module load intel./test < input.dat > output.dat 46

47 How to Submit a Job Use the command qsub to submit a job to the queuing system qsub takes a job submission script that contains special commands telling PBS what resources are needed It also contains the commands necessary to run the submitted job. Example: $qsub script.pbs To submit to a different queue: $qsub -q lab-long script.pbs 47

48 How to Check Job Status You can check job status using the showq command $showq -u username 48

49 Lab1: submit a very simple script Go to the scratch file system: $cd /scratch Create a directory: $mkdir myaccount Move to that directory: $cd myaccount Create a directory:$mkdir test Move to that directory: $cd test Copy file: $Cp ~cpsosa/gaussian/test/*. Submit your first script: $Qsub test.pbs 49

50 Example: Electronic Structure Calculation 50

51 Schrödinger Equation N-body wavefunction Energy of either the ground or an excited state of the system charges of the individual particles Spatial positions 51

52 Gaussian09 an Electronic Structure Suite of Programs Gaussian 09: is an electronic structure package capable of predicting many properties of atoms, molecules, and reactions, e.g. molecular energies and structures vibrational frequencies molecular orbitals and much more utilizing ab initio, density functional theory, semiempirical, molecular mechanics, and hybrid methods. 52

53 Gaussian History Gaussian70, Gaussian76, Gaussian77, Gaussian78, Gaussian80, Gaussian82, Gaussian83, Gaussian85, Gaussian86, Gaussian88, Gaussian90, Gaussian 92, Gaussian93, Gaussian 94, Gaussian95, Gaussian96, Gaussian 98, Gaussian 03, Gaussian 09 Born Died Nationality Fields Alma mater Doctoral advisor John Pople 31 October 1925 Burnham-on-Sea, Somerset, England March 15, 2004 (aged 78) Chicago, Illinois, United States England Theoretical chemistry Quantum chemistry Computational chemistry Cambridge University John Lennard-Jones Notable awards Nobel Prize in Chemistry (1998) 53

54 Input Format % Resource management # Route card Title section Molecular coordinates Geometric variables Other input options blank line blank line blank line blank line blank line 54

55 Input File %mem=32mb #p hf/6-31g opt hf/6-31g optimization of water 0 1 o h 1 oh h 1 oh 2 aoh oh=0.9 aoh=104.0 system resources computational model type of calculation title charge & multiplicity structure definition (z-matrix) variable values 55

56 Route Card Description: specifies keywords and options always begins with a # character keywords can be specified in any order options are grouped in parentheses, () keywords should not be repeated route section can be up to 5 lines long ended with a blank line Syntax: #[p] method/basis [keyword=(opt1,opt2)] [keyword2=(opt1[,opt2])] 56

57 Type of Calculation single point energy and properties geometry optimization reaction path following/searching frequency 57

58 Computational Method Level of theory: molecular mechanics amber, dreiding, uff semi-empirical am1, pm3, mndo, density functional theory b3lyp, mpwpw91, custom ab initio hf, mp2, ccsd, qcisd, hybrid g2, g3, oniom, 58

59 Lab3: Run Gaussian09 1. Copy files: $cp ~cpsosa/gaussian/g09/*. 2. Build your h2o.com input file 3. Build your h2o.pbs 4. Submit your h2o.pbs to the QUE 5. Visualize results 59

60 Lab 4: Visualize Results 1. Load the gaussian module if you do not have already loaded: $module load gaussian 2. Run GaussView by typing: $gview 60

61 Questions? MSI help desk is staffed Monday through Friday from 8:30AM to 7:00PM Walk-in help available in room 569 Walter Phone

62 Thank You! Many thanks to: Nancy Rowe for providing The MSI slides MSI Support