VPAC100: Introduction to High Performance Computing Using Linux

Transcription

1 VPAC100: Introduction to High Performance Computing Using Linux 1

2 Welcome to the VPAC training course. Moore's law, first observed by Gordon Moore of Fairchild Camera and Instrument Corp. in 1965, states that the numbers of transistors placed on a minimum cost integrated circuit doubles approximately every twenty-four months. Moore was discussing computing power from the previous six years from his observation, and yet it has remained largely true since than as well, and indeed has been extended to other metrics. For the user, it means we has meant presently we can take advantage of computer resources an order of magnitude roughly every four years. That, coupled along with parallel architecture and programming advances have allowed researchers to tackle simulations and data sets of enormous magnitude never before thought possible. Your attendance here today is a good sign for us that you wish to be part of that trend. The introduction course is designed to cater for a wide range of computer skill sets; covering basic concepts and command line interfaces for the absolute beginner, through to setting environment variables with modules and submitting jobs. It is certainly not necessary for the beginner to dwell too much on the advanced material, nor for the expert to suffer familiar territory. After working through the basic operations, job submission sections, and advanced sections users will be given the chance to work through a tutorial where they can submit a simple jobs to the cluster using a molecular dynamics program, NAMD, a finite element analysis, Abaqus, and a cloud-enabled submission for the statistical package, R. Each of these will outline and emphasise the basic workflow procedure for submitting high performance computing jobs. By attending the training course today, we hope that you are able to learn what VPAC High performance computing capabilities and services are, and able to take full advantage of them. We understand that for many of you, the computing front end can be a little intimidating. We are here to help! There is no such thing as a bad question, - a simple answer is all the difference from swirling in a dark whirlpool to seeing the proverbial light. Good luck & DON'T PANIC This manual was written by Lev Lafayette and Mike Kuiper,

3 Table of Contents 1.0 Introduction to VPAC Who We Are Creating an Account At VPAC VPAC's Hardware VPAC's Software Adding Licensed Software What Is An HPC System? How to Use RT 2.0 An Introduction to Linux Logging On Exploring the Environment Files and Editing Files Transferring Files Editing, Creating Directories, Moving Files Copying Directories, File Differences Searching for and within files, Wildcards Environment Modules and the Portable Batch System Environment Modules 34 Portable Batch System 38 Grid Australia HPC Tutorials NAMD and VMD Abaqus R and Grisu Further HPC Exercises Command Summary sheet. 3 60

4 1.0 An Introduction To VPAC Who We Are The Victorian Partnership for Advanced Computing (VPAC) is is a not for profit registered research agency established in 2000 by a consortium of Victorian Universities. VPAC's members include Deakin University, LaTrobe University, Monash University, RMIT University, Swinburne University of Technology, the University of Melbourne, the University of Ballarat and Victoria University. VPAC's purpose is to provide independent expert services, training, and support in advanced computing to its members. Person's employed or enrolled at any of the member Universities are entitled to have an account subject to approval by their University Operations Committee member. Such an account provides access to a number of supercomputers for High Performance Computing (HPC) in addition to technical support and training. Our activities are in a number of fields: High Performance Computing (HPC) Grid computing (lead agent of ARCS) Engineering (work with Holden, AutoCRC, etc) Life Sciences (our own pet molecular modeller) Geodynamics (with Monash and Caltech) Each Member has a share of the cycles, proportional to subscription. For users there is no charge for use. A share is also reserved for Industry. There is also extra capacity for VLSCI Stage 0. and access includes programming support and advice for use of the facility. What We Have (Hardware) For researchers, VPAC's internationally recognised HPC facility provides advanced computing tools and capabilities is managed by an expert team of software engineers and system administrators who provide high-level support. These supercomputers run a wide variety of compilers, scientific and mathematical programs, and libraries including: There are three clusters that VPAC manages, Wexstan, Edda and Tango. Each of these are Linux-based systems and are summarised as follows: 4

5 Wexstan 16 IBM e325 nodes bit CPUs, using 2 GHz Dual Opteron and Myrinet backbone interconnect, running CentOS 5 GNU/Linux as the operating system. 56 GB Ram and 0.5 Terabytes of Hard Disk Storage. Software on wexstan.vpac.org includes: abaqus fftw fluent gcc gmp hdf5 hyperworks intel iprscan java latentgold mesa mpfr namd openmpi perl petsc pgi python tcl wine Edda 188 CPU, 47 node Linux cluster, using Power5 1.67GHz CPUs running SuSE 9.0 GNU/Linux. Half the nodes have 16GB of RAM each and the other half 8GB each. Edda's benchmark performance is estimated to be at 1 teraflop. Software on edda.vpac.org includes: amber blender dock fds fftw gamess gcc gdal gmp gromacs ibmxlc ibmxlf imagemagick java lam lsdyna marmot mesa mpich mpj mrbayes namd ncarg neinastran openfoam openmpi perl petsc pgplot povray python scons srb stgermain underworld valgrind xmds Tango 760 CPUs, 92 Compute nodes, with Quad Core AMD Opteron GNU/Linux CPUs, using Infiniband interconnect (~2µs latency) and running 64-bit CentOS 5. Each node has 32GB RAM and four 320 GB disks. Software on tango.vpac.org includes: 5

6 Abaqus ACML AMBER ANSYS Ant Atlas Autodock AutoGrow binutils BLAST Blender BLT Bonnie++ Boost BWA CDAT CUDA DAWN DDD DOCK Ecat EGSnrc EM3D? EMBOSS FDS FFTW Fluent freeglut Freesurfer FSL Gamess GATE Gaussian GCC GD GDB GEANT GENREG GEOS Git GLUE GMP GotoBLAS Grace Graphviz GROMACS GSL H5utils Harminv HDF5 Hadoop HPMPI HyperWorks hypre IDL IMOD Intel Compilers Intel-mpi IPRSCAN ISP ITKSNAP JAIDA JAS3 JasPer Java lal lammps latentgold libctl libelf libfame libgdiplus libpng libsvm libtool ligplot lmf lp_solve lsdyna madymo maq marmot mash matio MATLAB meep meme mesa metaio mgltools minibaum molden molekel mono mopac mpfr mpiblast mpich mpiexec mrbayes mummer mvapich namd nauty neinastran netcdf ns octave openbabel openfoam openmpi padb pahole paraview pbsssh pcre perl petsc PGI compilers phaser povray python quilt R rosetta rysnc sabre schrodinger scilab scons semtex smem speccpu spinner srb stata svm-perf szlib tau tcl tk tkcon torque ultrascan underworld valgrind velvet visualdoc vmd wien2k wine xfoil xmds zlib What We Have (Software) Abaqus: A package for finite element analysis, usually applied in mechanical engineering. ABWT: The AB WT Analysis Pipeline is an off-instrument SOLiD data analysis software package for the analysis of experiments run. It maps reads from a transcript sample to a reference genome and assigning tag counts to features of the reference genome. ACML: The AMD Core Math Library (ACML) is a set of optimised and threaded math routines, especially useful for computational-intensive tasks. AMBER: Assisted Model Building with Energy Refinement (AMBER) is a family of force fields for molecular dynamics of biomolecules. AMBER is also the name for the molecular dynamics software package that simulates these force fields. ANSYS: ANSYS is an engineering simulation for general-purpose finite element analysis and computational fluid dynamics. Ant: Apache Ant as an automated software build processes, like Make etc, but is implemented using the Java language, requiring the Java platform, and best suited to building Java projects. 6

7 Atlas: Automatically Tuned Linear Algebra Software (ATLAS) is a software library for linear algebra, providing an open source implementation of BLAS APIs for C and Fortran77. Autodock: AutoDock is a suite of docking tools designed to predict how small molecules, such as substrates or drug candidates, bind to a receptor of known 3D structure. AutoGrow: AutoGrow uses AutoDock as the selection operator. For each generation, all ligand files are docked to the target protein, and for each dock, AutoDock returns a predicted binding affinity. AutoGrow (Java DOCK), uses fragment-based growing, docking, and evolutionary techniques. BEAM: BEAMnrc is a general purpose Monte Carlo simulation system for modelling radiotherapy sources which is based on the EGSnrcMP code system for modelling coupled electron and photon transport. binutils: The GNU Binary Utilities, or binutils, is a collection of programming tools for the manipulation of object code in various object file formats. They are typically used in conjunction with GNU Compiler Collection, make, and GDB. BLAST: The NCBI Basic Local Alignment Search Tool (BLAST) finds regions of local similarity between DNA sequences sequences and can be used to infer functional and evolutionary relationships between sequences as well as help identify members of gene families. Blender: Blender is a 3D graphics application used for modeling, UV unwrapping, texturing, rigging, water simulations, skinning, animating, rendering, particle, and other simulations, non-linear editing, compositing, and creating interactive 3D applications. The image (right) was created with Blender. BLT: The BLT library is an extension to Tcl/Tk. that adds plotting widgets, a geometry manager, a canvas item, and several new commands to Tk. Bonnie++: Bonnie++ is a benchmark suite that is aimed at performing a 7

8 number of simple tests of hard drive and file system performance. Boost: The Boost C++ libraries are a collection of open source libraries that extend the functionality of C++. They range from general-purpose libraries like the smart_ptr library to libraries primarily aimed at other library developers and advanced C++ users, like the metaprogramming template (MPL) and DSL creation (Proto). BWA: The Burrows-Wheeler Alignment (BWA) Tool is a fast light-weight tool that aligns short sequences to a sequence database, such as the human reference genome. Circuitscape: Circuitscape is a free, open-source program which borrows algorithms from electronic circuit theory to predict patterns of movement, gene flow, and genetic differentiation among plant and animal populations in heterogeneous landscapes. CDAT: The Climate Data Analysis Tools (CDAT) is a software infrastructure that uses Python. The CDAT subsystems, implemented as Python modules, provide access to and management of gridded data (Climate Data Management System or CDMS); large-array numerical operations (Numerical Python); and visualization (Visualization and Control System or VCS). The image (left) is a composite of CDAT windows. CLHEP: CLHEP (Class Library for High Energy Physics) is a C++ library that provides utility classes for general numerical programming, vector arithmetic, geometry, pseudorandom number generation, and linear algebra, specifically targeted for high energy physics simulation and analysis software. CPMD: The Car-Parrinello Molecular Dynamics code is a parallelized plane wave/pseudopotential implementation of Density Functional Theory, particularly designed for ab-initio molecular dynamics. CUDA: The NVIDIA CUDA Toolkit includes accelerated BLAS and FFT implementations, parallel thread execution and CUDA command line compiler. CUDA (an acronym for Compute Unified Device Architecture) is a parallel computing architecture developed by NVIDIA, the computing engine in NVIDIA graphics processing units (GPUs). DAWN: Drawer for Academic WritiNgs (DAWN) is a renderer which reads 3D geometrical data and visualizes with a vectorized 3D PostScript processor with analytical hidden line/surface removal. Desmond: Desmond is a software package developed at D.E. Shaw Research 8

9 to perform high-speed molecular dynamics simulations of biological systems on parallel systems that is available as part of Schrodinger or as a stand-alone package. DDD: Data Display Debugger (DDD) is a graphical front-end for command-line debuggers such as GDB, DBX, WDB, Ladebug, JDB, XDB, the Perl debugger, the bash debugger bashdb, the GNU Make debugger remake, or the Python debugger pydb. Besides typical features such as viewing source texts, DDD has also provides interactive graphical data display, where data structures are displayed as graphs. DOCK: DOCK simulates the problem of docking molecules to each other. In the field of molecular modeling, docking is a method which predicts the preferred orientation of one molecule to a second when bound to each other to form a stable complex. Ecat: Comprehensive C Library and utilities to handle Ecat, Interfile and Analyze datasets.allows conversion, and access to file internals. EGSnrc: EGSnrc is a package for the Monte Carlo simulation of coupled electron-photon transport. EM3D: EM3D is an integrated software application designed to facilitate the analysis and visualization of electron microscope (EM) tomography data by cellular and molecular biologists. EMBOSS: The European Molecular Biology Open Software Suite (EMBOSS) is a molecular biology tool which copes with data in a variety of formats and even allows transparent retrieval of sequence data from the web. FDS: Fire Dynamics Simulator (FDS) is a computational fluid dynamics model of fire-driven fluid flow. The software solves numerically a form of the NavierStokes equations appropriate for low-speed, thermally-driven flow, with an emphasis on smoke and heat transport from fires. FFTW: "Fastest Fourier Transform in the West" (FFTW) is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sine transforms or DCT/DST). FLUENT: FLUENT is a flexible general-purpose computational fluid dynamics package used for engineering sj20.15e.o imulations of all levels of complexity. freeglut: freeglut is a completely opensource alternative to the OpenGL Utility Toolkit (GLUT) library. 9

10 FreeSurfer: FreeSurfer is a set of automated tools for reconstruction of the brain s cortical surface from structural MRI data, and overlay of functional MRI data onto the reconstructed surface. The image (left) is from the Surfstat page, a MATLAB toolbox that works with FreeSurfer. FSL: FSL is a comprehensive library of analysis tools for FMRI, MRI and DTI brain imaging data. G4beamline: G4Beamline is a particle tracking and simulation program based on the Geant4 toolkit that is specifically designed to easily simulate beamlines and other systems using single-particle tracking. GAMESS: General Atomic and Molecular Electronic Structure System (GAMESS) is a program for ab initio molecular quantum chemistry. A variety of molecular properties, ranging from simple dipole moments to frequency dependent hyperpolarizabilities may be computed. GATE: The Geant4 Application for Emission Tomography (GATE) provides comprehensive physics modeling abilities of the general purpose codes while making it possible to intuitively configurate an Emission Tomography simulation. GATE allows the accurate description of time-dependent phenomena such as source or detector movement and source decay kinetics. Gaussian: Gaussian provides electronic structure modeling which can be applied to both stable species and compounds which are difficult or impossible to observe experimentally. Gaussian can be used for comprehensive investigations of molecules and reactions, predicting and interpreting spectra, exploring diverse chemical arenas, and complex modelling. GCC: The GNU Compiler Collection (GCC) is a compiler system produced by the GNU Project supporting various programming languages including C and C++ with front ends for Fortran, Pascal, Objective-C, Java, Ada and others. GD: The Graphics Draw (GD) Library is a graphics software library for dynamically manipulating images. Its native programming language is ANSI C, but it has interfaces for many other programming languages. GDB: The GNU Debugger (GDB) is the standard debugger for the GNU software system. It is a portable debugger that runs on many Unix-like 10

11 systems and works for many programming languages, including Ada, C, C++, FreeBASIC, FreePascal and Fortran. GEANT: GEometry ANd Tracking (GEANT) is a simulation software designed to describe the passage of elementary particles through matter, using Monte Carlo methods. GENREG: Generator fuer regulaere Graphen (GENREG) generates regular graphs for the chosen parameters and constructs them. GEOS: GEOS (Geometry Engine - Open Source) is a C++ port of the Java Topology Suite (JTS). As such, it aims to contain the complete functionality of JTS in C++. This includes all the OpenGIS Simple Features for SQL spatial predicate functions and spatial operators, as well as specific JTS enhanced topology functions. Git: Global information tracker (Git) is a fast, scalable, distributed revision control system with an unusually rich command set that provides both highlevel operations and full access to internals. GLUE: Grid LSC User Environment (GLUE) is a collection of utilities for running data analysis pipelines for online and offline analysis as well as accessing various grid utilities. It also provides the infrastructure for the segment database. GMP: GNU MP (GMP) is a library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating point numbers. It has a rich set of functions, and the functions have a regular interface. It is particularly designed for speed. GotoBLAS: The GotoBLAS codes are a fast implementation of the Basic Linear Algebra Subroutines. The advantage is fast calculation which makes use of all instruction sets of modern processors. Grace: Grace is a tool to make two-dimensional plots of numerical data. It combines the convenience of a graphical user interface with the power of a scripting language which enables it to do sophisticated calculations or perform automated tasks. Graphviz: Graphviz is open source graph visualization software with several main graph layout programs, interactive graphical interfaces, auxiliary tools, libraries, and language bindings. GROMACS: The GROningen MAchine for Chemical Simulations (GROMACS) is a molecular dynamics simulation package that is very fast and has support for different force fields. It is notable for being used for protein folding at Folding@Home. The image (right, below) is Gromacs in action. 11

12 GSL: The GNU Scientific Library (GSL), is a collection of numerical routines for scientific computing. The library provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. H5utils: The package h5utils is a set of utilities for visualization and conversion of scientific data in the free, portable HDF5 format along with programs to convert HDF5 datasets into the formats required by other free visualization software (e.g. plain text, Vis5d, and VTK). Harminv: Harminv is used to solve problems of harmonic inversion - given a discrete-time, finite-length signal that consists of a sum of finitely-many sinusoids (possibly exponentially decaying) in a given bandwidth, it determines the frequencies, decay constants, amplitudes, and phases of those sinusoids. It can, in principle, provide much better accuracy than straightforwardly extracting FFT peaks. HDF5: HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. Hadoop (module hod): Apache Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. HPMPI: HP-MPI is a high performance and production quality implementation of the Message-Passing Interface (MPI) standard for HP servers and workstations. HyperWorks: Altair HyperWorks is a computer-aided engineering simulation software platform that includes modeling, analysis, visualization and data management solutions for linear, nonlinear, structural optimization, fluidstructure interaction, and multi-body dynamics applications. hypre: Hypre is a library of high performance preconditioners that features parallel multigrid methods for both structured and unstructured grid problems. 12

13 IDL: Interactive Data Language (IDL) is a programming language used for data analysis. IDL is vectorized, numerical, and interactive, and is commonly used for interactive processing of large amounts of data (including image processing). IMOD: IMOD is a set of image processing, modeling and display programs used for tomographic reconstruction and for 3D reconstruction of EM serial sections and optical sections. IMSL: (International Mathematics and Statistics Library) is a commercial collection of software libraries of numerical analysis functionality that are implemented in the computer programming languages of C, Java, C#.NET, and Fortran. Intel Compilers: Intel compilers are optimised for its hardware platforms to minimise stalls and produce code that executes in the fewest number of cycles. Intel's suite of compilers has front ends for C, C++, and Fortran. Intel-mpi: Intel's MPI Library and runtime environment for Linux. IPRSCAN: IntroProScan (iprscan) is a tool that combines different protein signature recognition methods into one resource. InterProScan is more than a simple wrapping of sequence analysis applications since it requires performing a considerable data look-ups from some databases and program outputs. ISP: In-situ Partial Order (ISP) is a dynamic verifier for MPI Programs. ISP will help you debug your programs, and graphically show you all the possible send/receive matches, barrier synchronizations, etc. ITKSNAP: ITK-SNAP is used to segment structures in 3D medical images, providing semi-automatic segmentation using active contour methods, as well as manual delineation and image navigation. JAIDA: JAIDA is a Java (J) implementation of the Abstract Interfaces for Data Analysis (AIDA). JAIDA allows Java programmers to create histograms, scatter plots and tuples, perform fits, view plots and store and retrieve analysis objects from files. JAS3: JAS3 is a follow on from Java Analysis Studio (JAS), a general purpose data analysis tool for histograms, XY plots, scatterplots, export of plots in a variety of formats, and AIDA compliant analysis system. JasPer: JasPer is a collection of software (i.e., a library and application programs) for the coding and manipulation of images. Java: The Java programming language is an object-orientated language similar to C and C++ in syntax, but with a simpler object model and fewer low-level facilities. It is designed for developing cross-playment applications. 13

14 LAL: The LSC Algorithm Library (LAL) is a collection of routines written in ANSI C99 for gravitational wave data analysis. LAMMPS: Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) has potentials for soft materials and solid-state materials (metals, semiconductors) and coarsegrained or mesoscopic systems. It can be used to model atoms or as a parallel particle simulator at the atomic, meso, or continuum scale. The image (left) is atom-to-continuum coupling using LAMMPS. LatentGOLD: Latent GOLD is a latent class and finite mixture program. Latent GOLD contains separate modules for estimating three different model structures; LC Cluster models, DFactor models, and LC Regression models. libctl: libctl is a free Guile-based library implementing flexible control files written originally to support our Photonic Bands and Meep software. libelf: LibELF lets you read, modify or create Executable and Linkage Format (ELF) files in an architecture-independent way. The library takes care of size and endian issues. libfame: libfame is a real-time MPEG-1 and MPEG-4 rectangular and arbitrary shaped video encoding library. libframe: libframe is a toolkit that contains various tools useful for development in C++, ranging from a Config class to an Expression library, to a set of abstract tuple handling classes, to an event handling application frame work with configurable thread pooling. libgdiplus: Libgdiplus is the Mono library that provide a GDI+ comptible API on non-windows operating systems. libpng: libpng is a Portable Network Graphics (PNG) reference library. Portable Network Graphics (PNG) is a bitmapped image format with lossless data compression. libsvm: libsvm is a library for Support Vector Machines (SVM). SVMs are a set of related supervised learning methods used for classification and regression. 14

15 libtool: GNU libtool is a generic library support script. Libtool hides the complexity of using shared libraries behind a consistent, portable interface by encapsulating both the platform-specific dependencies, and the user interface, in a single script. ligplot: Ligplot Is a program for automatically generating schematic diagrams of protein-ligand interactions for a given PDB file. LMF: The Local Maximum Fitting (LMF) algorithm firstly finds local maximums within a certain time window, and regenerate the time series data as a sum of harmonic curves. The number of harmonic curves is limited by AIC ( Akaike Information Criterion ) to avoid over-fitting. LP_SOLVE: LP_SOLVE is a linear programming code written in ANSI C, which has solved problems as large as 30,000 variables and 50,000 constraints. Lp_solve can also handle (smaller) integer and mixed-integer problems. LSDYNA: LS-DYNA is general-purpose multiphysics simulation software package typically used for highly nonlinear transient dynamic finite element analysis (FEA) using explicit time integration. MADYMO: MAthematical DYnamic MOdels (MADYMO) is a multibody dynamics solver and frequently used for automobile occupant safety/injury calculations. Maq: Maq builds mapping assemblies from short reads generated by the nextgeneration sequencing machines. It is particularly designed for IlluminaSolexa 1G Genetic Analyzer, and has preliminary functions to handle ABI SOLiD data. Marmot: Marmot is a library written in C++, which has to be linked to your application in addition to the existing MPI library. It will check if your application conforms to the MPI standard and will issue warnings if there are errors or non-portable constructs. Mash: Mash is a toolkit for multimedia using IP multicast. The Mash toolkit is an outgrowth of the MBone tools (e.g.--sdr, vic, vat) developed to support streaming audio and video applications. matio: Libmatio is an open-source library for reading/writing Matlab MAT files. This library is designed for use by programs/libraries that do not have access or do not want to rely on Matlab's libmat shared library. 15

16 MATLAB: MATLAB DCS is a numerical computing environment allowing matrix manipulation, plotting of functions and data, implementation of algorithms etc. on a cluster. The image (right) is the MATLAB desktop environment. MEEP: MIT Electromagnetic Equation Propagation (MEEP) is a finite-difference time-domain (FDTD) simulation software package developed at MIT to model electromagnetic systems. MEME: MEME provides tools for discovering and using protein and DNA sequence motifs, a pattern of nucleotides or amino acids that appears repeatedly in a group of related DNA or protein sequences. Mesa: Mesa is an implementation of the OpenGL specification for rendering interactive 3D graphics, usable in a variety of environments from software emulation to GPUs. Metaio: Metaio contains a library for parsing LIGO_LW Table files and can read XML files compressed with the gzip compression algorithm. MGLTools: Developmed by the Molecular Graphics Laboratory (MGL) MGLTools, is used for visualization and analysis of molecular structures. It includes AutoDockTools (ADT), Python Molecular Viewer (PMV) and Vision, a visual-based programming environment. Minibaum: Minibaum3 is a small C program which has been used for hypohamilton graphs and angular momentum graphs. Molden: Molden displays molecular density from the ab initio packages GAMESS and GAUSSIAN and others. Molden reads all the required information from the GAMESS / GAUSSIAN outputfile. Molden is capable of displaying molecular orbitals, electron density and molecular minus atomic density. Molekel: Molekel is a molecular visualization program that import and exports data using OpenBabel and displays molecules with different rendering styles, generates iosurfaces, and animates. Mono: Mono is used to develop an Ecma standard compliant,.net-compatible 16

17 set of tools, including a C# compiler and a Common Language Runtime. MOPAC: Molecular Orbital PACkage (MOPAC) is a semiempirical quantum chemistry program based on Dewar and Thiel's NDDO approximation. MPFR: The GNU MPFR library is a C library for multiple-precision floatingpoint computations with correct rounding. mpiblast: mpiblast is an implementation of the bioinformatics software NCBI BLAST, which finds regions of local similarity between sequences. Through database fragmentation, query segmentation, intelligent scheduling, and parallel I/O, it improves performance by several orders of magnitude. MPICH: MPICH is a free and portable implementation of MPI, a standard for message-passing for distributed-memory applications used in parallel computing. Mpiexec: Mpiexec is a replacement program for the script mpirun, which is part of the mpich package. It is used to initialize a parallel job from within a PBS batch or interactive environment. MrBayes: MrBayes conducts Bayesian estimation of phylogeny based on the posterior probability distribution of trees, which is the probability of a tree conditioned on the observations. MUMmer: MUMmer is rapidly aligns entire genomes, including incomplete genomes and contigs from a shotgun sequenceing program. MVAPICH: MVAPICH implements MPI over InfiniBand, 10GigE/iWARP and RDMA over Ethernet. NAMD: NAnoscale Molecular Dynamics (NAMD) is a molecular dynamics simulation package written using the Charm++ parallel programming model, often used to simulate large systems (e.g., millions of atoms). nauty: nauty is a program for computing automorphism groups of graphs, digraphs and can also produce a canonical labelling. NEI Nastran: NEi Nastran is a finite element analysis (FEA) solver used to generate solutions for linear 17

18 and nonlinear stress, dynamics, and heat transfer characteristics of structures and mechanical components. The image (right, above) is a Nastran model. netcdf: The Unidata network Common Data Form (netcdf) is an interface for scientific data access and a library that provides an implementation of the interface. The netcdf library also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data. NS: Network Simulator (NS) is a discrete event simulator targeted at networking research. Ns provides substantial support for simulation of TCP, routing, and multicast protocols over wired and wireless (local and satellite) networks. Octave: GNU Octave is a high-level language, primarily intended for numerical computations and is highly compatiable with MATLAB. Open Babel: Open Babel is a chemical toolbox which can read, write and convert over 90 chemical file formats, and filter and search molecular files using SMARTS and other methods. OpenFOAM: Open Field Operation and Manipulation (OpenFOAM) is primarily a C++ toolbox for the customisation and extension of numerical solvers for continuum mechanics problems, including computational fluid dynamics. Open MPI: Open MPI combines the merger of three major MPI implementations (FT-MPI, LA-MPI, and LAM/MPI) to create a complete MPI-2 implementation. Padb: Padb works as a parallel front end to gdb allowing it to target parallel applications. pahole: Analyzes your code and identifies unused memory holes in data structures, and suggests re-ordering to improve memory usage and speed. Paraview: ParaView is a data analysis and visualisation application. The data exploration can be done interactively in 3D or programmatically using ParaView's batch processing capabilities. ParaView was developed to analyze extremely large datasets. PBSssh: PBSssh: Is a Bourne-Again shell executable for Portable Batch Script. PCRE: Perl-Compatible Regular Expression (PCRE) library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. Perl: Perl is a high-level, general-purpose, interpreted, dynamic programming 18

19 language designed as a a general-purpose Unix scripting language to make report processing easier. PETSc: PETSc is a suite of data structures and routines for solution of scientific applications modeled by partial differential equations. PGI compilers: PGI compilers are a set of Fortran, C and C++ compilers for High Performance Computing Systems from Portland Group. Phaser: Phaser consists of CCP4 and BLT. The former is used to determine macromolecular structures by X-ray crystallography, and other biophysical techniques. BLT has ben previously described. POV-Ray: Persistence of Vision Raytracer (POV-RAY) is a high-quality for creating three-dimensional graphics. Features include radiosity, photon mapping, focal blur, and other photorealistic capabilities. Python: Python is a general-purpose high-level programming language that aims for high levels of readability. It features a fully dynamic type system and automatic memory management. Like other dynamic languages it is often used as a scripting language. Quilt: Quilt is a tool to manage large sets of patches by keeping track of the changes each patch makes. Patches can be applied, unapplied, refreshed, etc. R: R is a programming language and software environment for statistical computing, where it is a defacto standard, and graphics. Rosetta: Rosetta is a molecular modeling software package for understanding protein structures, protein design, protein docking, protein-dna and proteinprotein interactions. rysnc: Rsync copies files either to or from a remote host, or locally on the current host. It is quick because it only copies changed files. SABRE: Software for the Analysis of Recurrent Events (SABRE) is a program for the statistical analysis of multi-process random effect response data. These responses can take the form of binary, ordinal, count and linear recurrent events. Schrodinger: Jaguar is an ab initio quantum chemistry package for both gas and solution phase calculations, with strength in treating metal-containing systems produced by the Schrödinger company. Scilab: Scilab is a scientific software package for numerical computations which includes hundreds of mathematical functions, sophisticated data structures (including lists, polynomials, rational functions, linear systems), an interpreter and a high level programming language. 19

20 SCons: SCons is a software construction tool (build tool, or make tool) implemented in Python, that uses Python scripts as "configuration files" for software builds. Semtex: Semtex is a family of spectral element simulation codes. The spectral element method is a high-order finite element technique that combines the geometric flexibility of finite elements with the high accuracy of spectral methods. smem: smem is a memory reporting tool, notable for its ability to report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. SPEC CPU2006: The Standard Performance Evaluation Corporation (SPEC) that has a standard set of relevant benchmarks for computer systems. SPEC CPU2006 measures the performance of the processor, memory architecture, and compilers. Spinner: Spinner is an anti-idle program that displays a little "spinning" ASCII character in the top left corner of your terminal. Spinner is useful for keeping ssh links from dropping due to inactivity. SRB: Storage Resource Broker (SRB) is a Data Grid Management System (DGMS) or simply a logical distributed file system based on a client-server architecture which presents the user with a single global logical namespace or file hierarchy. Stata: Stata is a integrated statistical package that provides data analysis, data management, and graphics that includes linear mixed models, multivariate methods, multinominal probit and Mata, a matrix language. SVMperf: Support Vector Machine for Multivariate Performance Measures (SVMperf) is an implementation of the Support Vector Machine (SVM) formulation for optimizing multivariate performance measures and implements alternative structural formulation of the SVM optimization problem for conventional binary classification with error rate and ordinal regression. Szlib: Szip is a freeware portable general purpose lossless compression program. TAU: Tuning and Analysis Utilities (TAU) is a program and performance analysis tool for high-performance parallel and distributed computing with a suite of tools for static and dynamic analysis of programs written in C, C++, FORTRAN 77/90, Python, High Performance FORTRAN, and Java. Tcl: Tool command language (Tcl) is a scripting language commonly used for rapid prototyping, scripted applications, GUIs and testing. 20

21 Tk: Tk is an a library of basic elements ("widgits") for building a graphical user interface. It is typically used with Tcl. tkcon: tkcon is a replacement for the standard console that comes with Tk which provides many more features than the standard console. TORQUE: Terascale Open-Source Resource and QUEue Manager (TORQUE) is a distributed resoource manager with notable fault tolerance, scalability and a useful scheduling interface. UltraScan: UltraScan is used for the analysis of ultracentrifugation data. The software features an integrated data editing and analysis environment in graphical user interface, popular sedimentation and equilibrium analysis methods with support for velocity and equilibrium experiments, single and multi-channel centerpieces, absorbance and interference optics. Underworld: Underworld is a 3D-parallel geodynamic modelling framework capable of deriving viscous / viscoplastic thermal, chemical and thermochemical models consistent with tectonic processes, such as mantle convection and lithospheric deformation over long time scales. Valgrind: Valgrind is an instrumentation framework for building dynamic analysis tools. Velvet: Velvet is a set of algorithms manipulating de Bruijn graphs for genomic Sequence assembly. It was designed for short read sequencing technologies, such as Solexa or 454 Sequencing. The tool takes in short read sequences, removes errors then produces high quality unique contigs. VisualDOC: VisualDOC is a general-purpose optimization tool that allows the user to quickly add design optimization capabilities to almost any analysis program. VMD: Visual molecular dynamics (VMD) is a molecular modelling and visualization computer program. VMD is primarily developed as a tool for viewing and analyzing the results of molecular dynamics simulations. WIEN2k: WIEN2k performs quantum mechanical calculations on periodic solids. It uses the full-potential (linearized) augmented plane-wave and localorbitals basis set to solve the Kohn Sham equations of density functional theory. Wine: Wine Is Not an Emulator (Wine) allows Unix-like computer operating systems to execute programs written for Microsoft Windows. XFOIL: XFOIL is an interactive program for the design and analysis of subsonic isolated airfoils. 21

22 XMDS: extensible Multi Dimensional Simulator (XMDS) is a numerical simulation package that integrates equation that converts XML files to a C++ program that integrates the equations. zlib: zlib is a general purpose data compression library with data formats specified by RFCs 1950 to Adding Licensed Software Additional software can be installed at users request depending on software licensing and availability. What Is A HPC System? Why Clusters? Why Linux? Why the Command Line? High-performance computing (HPC) is the use of supercomputers and clusters to solve advanced computation problems. A supercomputer is a nebulous term for computer that is at the frontline of current processing capacity, particularly speed of calculation. In contemporary machines this is measured in tera- and peta- FLOPS (floating point operations per second). One type of supercomputer architecture are clustered computers. Simply put, there are a collection of smaller computers strapped together with tha highspeed locla network. Applications can be parallelised across them through programming. Clustered computing is when two or more computers serve a single resource. This improves performance and provides redundancy in case of failure system. Parallel computing refers to the submission of jobs or processes over one or more processors and by splitting up the task between them. By the way of analogy consider a horse and cart as computer system and the load as the computing tasks. If one wants to move a greater load there is essentially three options. Re-arrange the load so it is more efficiently arranged. This is analogous to improving the code. It can help, and help significantly, but its ultimately limited. Purchase a bigger cart and a bigger horse to move the load. This is analogous to buying a bigger computer and getting better software. In computing, this rapidly develops decreasing returns. Distribute the load among several carts and horses, managed by a teamster. This is analogous to parallel processing in a cluster. It is the most cost-efficient and most scalable method. 22

23 VPAC is vendor-neutral, meaning that we use the best technology for the job. The best operating system technology for high performance, clustered systems and parallel computing is a UNIX-like operating system such as GNU/Linux. The reasons for this is are manifold. Firstly, GNU/Linux scales and does so with stability and efficiency. Secondly, critical software such as the Message Parsing Interface (MPI) and nearly all scientific programs are designed to work with GNU/Linux. Thirdly, the operating system and many applications are provided as "free and open source", which means that not only are there are some financial savings, were also much better placed to improve, optimize and maintain specific programs. Finally, there is the command line. For most users a Graphic User Interface (GUI) is how they interact with a computer system, and there are some advantages with this, not the least being a usually intuitive visual representation for actions. However this takes up significant computer resources. In contrast a command-line interface provides a great deal more power and is very resource efficient. Running supercomputers with a GUI is not a sound policy. These reasons correlate with actual application in the real world. In November 2009 of the Top 500 Supercomputers worldwide, only about 1% did not use a "UNIX-like" operating system (a decline from 1.4% on November 2007) and nearly all use an operating system that is entirely "free and open source" (a small percentage use a combination of free and proprietary systems). It is essential therefore, to become familiar with Linux and the command-line if one want to use High Performance Computing. 23

24 2.0 An Introduction to Linux In this introduction to Linux part of the course we will engage in several tasks. The first will be to log into a Linux system and familiarise ourselves with the environment. We will then create some files on the local machine and copy those files to the supercomputer. We will then log on to the supercomputer, modify those files and copy them back the local computer. Back on the local computer we'll create a directory, move the files to that directory and run some very basic search functions. When we want you to enter a command or follow a menu path, the font will be in Courier 10 pitch with a grey background. 1. Logging On From the Main Menu bar select Application > System Tools > Terminal. This will open a command line interface. When that is open, go the terminal menu and select File > Open Terminal.You should now have two terminal windos open. In the first terminal window we'll explore some of the basic commands on the local machine In the second terminal window we'll do the same, but on the supercomputer. To log on to a VPAC supercomputer, you will need a user account and password and a Secure Shell 2 (ssh) client. VPAC does not allow protocols such as Telnet, FTP or RSH as they insecurely send passwords in plain-text over the network. Linux distributions almost always include SSH as part of the default installation as does Mac OS 10.x, although you may also wish to use the Fugu SSH client. For MS-Windows users, we recommend using the PuTTY SSH client. If using Mac OS 10.x, you will probably want to add a terminal alias to your dock. From the Macintosh HDD and go to the Applications folder, then Utilities from within that. Terminal is in the Utilities folder. Drag it to an empty space in the Dock, and the operating system will put an alias there. If you are using a graphic interface for Linux, like GNOME or KDE, you may wish to do the same with one of the terminal clients and panels. If you're using MS-Windows, download PuTTY. In the 'Host Name' box, enter the server you want to connect to (e.g., tango.vpac.org) and select SSH from the 'Connection type' radio button. Verify the host key when connecting for the first time. You will also probably want to have X-forwarding enabled for any connections that require graphic forwarding. See the following image: 24

25 In the 'Host Name' box, enter the server you want to connect to (e.g., tango.vpac.org) and select ssh from the 'Connection type' radio button. Its useful to enter a session name, "Tango" in the above case and save it so you don't need to remember the details next time. Generally, the other Putty settings will be fine as they are. One thing you might need if you are going to be using XWindows (to display a graphical interface from VPAC on your desktop) is to turn on XForwarding. You will also need some sort of "XWindows Server" installed on your desktop, perhaps XWin32 or Exceed3D. A possible free option is XMing, With Mac or Linux simply open the terminal client and enter your username and the machine you wish to connect to, followed by the password when connected. For example; ssh <your username>@tango.vpac.org Secure shell opens a connection to a remote system. If you want to enable graphic enabling you can use the -X or -Y (secure) options e.g., ssh X <your username>@tango.vpac.org 25

26 In our training course we are using Linux machines with the Fedora distribution installed. The ssh command above can be entered as it is written, replacing <your username> with the account you have been provided (train01, train02 etc). 2. Exploring The Environment The first thing we'll do is explore the environment of the command-line on both our local machine and the supercomputer. On both these systems, run the following commands. whoami "Who Am I?; prints the effective user id. pwd "Print working directory"; prints the directory where you're currently in. When a user logs in on a Linux or other UNIX-like system on the command line, they start in their home directory. The output of the above command should be: /home/<username> Now let's run a listing for the directory on both the local computer and the supercomputer: ls "List"; lists contents for particular directory, the current directory by default. Linux commands often come with options expressed as: <command> <option[s]> Run this command on both the local computer and the supercomputer. ls lart "List" with long format, including file permissions (l), include hidden files ('a', for all), sorted by reverse order ('r'), by modification time ('t'). Linux also have very useful 'pipes' and redirect commands. To pipe one command through another use the ' ' symbol. The who command how who is currently logged into the system. You may suspect that this will differ on the supercomputer and the local system! Run the command on the local computer and then 'pipe' the who command through 26

27 Run this command on the supercomputer. who u less "Who" shows who is logged on, how long they've been idle and piped through the less command. Another environment feature to explore is the ps or process status command. A number of programs can be run by a one or more users simultaneously, including helping programs called daemons. If no options are added ps selects all processes with the same effective user ID (euid=euid) as the current user and associated with the same terminal as the invoker. To see what is running, who is running it, the process ID, and how much CPU they are using use: ps afux less "ps" provides a list of current processes. The 'a' option list the processes of all users, the 'f' shows job hierarchy, the 'u' option provides additional information about each process, and the 'x' option includes non-user programs such as daemons. This is piped through less. To redirect output use the '>' symbol. To redirect input (for example, to feed data to a command) use the '<'. Concatenation is achieved through the use of '>>' symbol. Run this command on the supercomputer. w > list.txt The command 'w' acts like a combination of who, uptime and ps -a. This is redirected to the file list.txt This command lists the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes and redirects this information to the file list.txt Run the ls command to make sure you have the file list.txt on both the supercomputer and concatenate the file and print on the standard output (i.e., the terminal) using the cat command to ensure that the data from the who command is there. ls cat list.txt 27

28 3. Files and Editing Files Linux expresses its files as words made up of pretty much any characters, excepting the slash (/) which is used for directory navigation. In general however it is a good idea to avoid filenames with punctuation marks, non-printing characters (including spaces) as these can lead to some difficulties and annoyances, especially on the command-line level. It is a convention to use underscores instead of spaces e.g., this_is_a_long_name.txt Linux is case-sensitive with its filenames. This means that that list.txt is a different file to LIST.TXT, or even list.txt. Files do not usually require a program association suffix (the C compiler is an exception, for example), although you may find this convinient. The file list can be opened by a texteditor just as easily as list.txt. There are three text editors usually available on Linux systems. The first is nano, a very easy to use clone from the Pine client that uses control keys with a the equivalent of a "shortcut bar". The hefty EMacs (Editor Macros) editor and environment is a feature-rich program that was first written in 1976 and is now up to version 22.1! Also from 1976 is Vim (Vi improved) which is a series of enhancements build on the "screen orientated" text editor vi. Vim is generally understood to be a modal editor, operating either in a insert mode (where typed text becomes part of the document) or command mode (where keystrokes are interpreted as commands that control the edit session). Vi or Vim are often installed as the default text editor. In "UNIX culture" EMacs and Vim are considered favourites among experienced users, with nano considered the best for beginners. There are also long-running, and largely tongue-in-cheek, "editor wars" with various proponents debating the relative merits of different editors. Today we will use the nano editor to make some changes to the file list.txt which is on the supercomputer. To open the file for editing simply type: nano list.txt Use Cntrl-V or the cursor keys to navigate to the bottom of the page. Enter a blank line, then add these words of literary wisdom from Theodor Seuss Geisel. One fish, two fish, red fish, blue fish Then enter Cntrl-O to "write out" the file and Cntrl-X to "exit". 28

29 Nano; A Simple Text Editor Nano is an easy to use text editor typically available on Linux systems. Nano, a very easy to use clone from the Pine client that uses control keys with a the equivalent of a "shortcut bar". With nano editing is very intuitive. Start with nano <filename> on the command prompt. One can type straight to the display and editing is a simple function of simultaneously using Ctrl and a keystroke. The most commonly used key combinations are available on the bottom of the screen, including cutting (^K) and pasting ("uncutting", ^U) lines of text, searching ("where is", ^W), opening ("read a file", ^R), saving files ("write out", ^O), scrolling up and down the text (^Y, ^V). Further commands can be displayed through invoking help ("get help", ^G) such as search and replace (M-R, ie., meta key, usually Esc and 'R'). 4. Transferring Files in General To move files to and from the supercomputer and one's desktop you need to use an SCP (secure copy protocol) or SFTP (secure file transfer protocol) over SSH. If you are using Linux or Mac, you will be able to do this with the standard command-line interface with the general procedure of; scp source destination This however doesn't quite give the full story. Both the source or the destination may include a username and address, although if one is running the command the source or destination machine the account information does not have to be entered for that machine. Often a path to the files and 29

30 directories will be required as well. However remembering the order source then destination is good shorthand. The following is a more elaborate version of scp: scp source.address:/path/to/source destination.address:/path/to/destination/ If you are using MSWindows, we recommend using WinSCP, or, if using certain applications such as MATLAB, the PuTTY Secure Copy client PSCP. WinSCP comes with a intuitive GUI that provides basic file management functionality in addition to Secure Shell and Secure Copy functions. Linux and Mac users can also use a GUI for secure file transfers. For Linux users this is typically inbuilt with the file browser application. For Mac users you might wish to consider Fugu or Cyberduck (links below). Transferring files With Rsync and Unison RSync provides a way to keep two repositories of files "in sync", one of these repositories may be on your desktop, the other your home directory at VPAC. The nice feature of rsync is that it is very fast - after the initial backup. The reason for this is that it tracks changes. There is no point copying and rewriting an entire file when only a handful of characters have changed. The following is the basic command for rsync between two Linux machines: rsync av e ssh source/ username@remotemachine.com:/path/to/destination/ The -av ensures that it is in archive mode (recursive, copies symlinks, preserves permissions) and is verbose. The -e is to specify a transfer protocol, in this case ssh. Note that rsync is "trailing slash sensitive". A trailing / on a source means "copy the contents of this directory". Without a trailing slash it means "copy the directory". A tutorial on using rsync as a backup tool can be found at and some hints on 30

31 using rsync for MS-Windows at An alternative to rsync is Unison, maintained by Benjamin C. Pierce, professor in the Department of Computer and Information Science at the University of Pennsylvania. It is available for Linux, Mac and MS-Windows and can be operated through a GUI or with command-line tools. The main technical difference is that it is a two-way synchronisation tool, whereas rsync effectively offers one-way mirroring. For example, if you use rsync to mirror a file from the cluster to your desktop, then modify that file on the desktop, you risk ovewriting the changes you made the next time you conduct an rsync. Unison can work out what has changed, keep different versions and even merge the changes. The basic command line operation is achieved in a very similar syntax: unison a.tmp ssh://remotehostname/a.tmp The main disadvantage with Unison is that is not as widely deployed as rsync and requires installation on local and remote machines, and preferably of the same version. Putty is available from: WinSCP is available from: Fugu is available from: For training manual SCP client for Mac. More information on OpenSSH and the latest version can be found at: A tutorial on using rsync as a backup tool Rsync for MS-Windows Unison for Linux, Mac and MS-Windows 5. Transferring Files from the Supercomputer to Local We are going to copy the file list.txt from the supercomputer to the local machine. From the local machine enter the following command: 31

32 scp Be sure to replace <username> with your username on the supercomputer (train01, train02, train03 etc). When the transfer is complete check on the local machine that the file has transferred with ls. Two questions might come to mind when entering the command; firstly, why aren't we running the command on the supercomputer (a "put", rather than a "get") and secondly, what is the "." for? To answer the first question, it must be remembered that in order for a copy of files to occur, both machines have to know where the other one is, translated from hostnames and Internet Protocol (IP) addresses. In most cases, local machines use private IP addresses, not public addresses. For example, if you were on the supercomputer and wanted to copy the file to a local machine you might think the following could work, as it follows the suggested format of source and destination and uses the correct command: scp list.txt <username>@ The problem is, which of the multitudes of computers out there with a private address of do you want to copy list.txt to? How would the supercomputer know which machines have this private addresses? Even if it could find out, it would take a very long time to connect to all the switches in the world to find out! Explaining the '.' is a little easier. It simply refers to the current directory. Thus in the example, the source is tango, the destination is the directory the command is being run in. One could use the command cd. which would mean 'change directory to the directory you are currently in', which is pretty pointless. More useful however is cd.. which means change directory to the parent of the current directory. 6. Editing list.txt, Creating Directories, Moving Files We know have list.txt on the local computer. Let's add some new material to it before sending it back to the supercomputer. On the local computer enter: nano list.txt Navigate to the bottom of the file using Cntrl-V or the cursor keys and enter the following lines: This one has a little star. This one has a little car. Say! What a lot of fish there are. 32

33 Then write out the file (Cntrl-O) and exit (Cntrl-X). Our next step will be to create a directory to put this file in and then move the file into that directory. The navigate to the directory and make sure that it is there. On the local computer enter: mkdir seuss mv list.txt seuss/ cd seuss ls pwd The output should be the list.txt file from the ls command and /home/<user>/seuss from the pwd command. 7. Copying Directories, File Differences The next step will be to copy the directory and its contents from the local computer to the supercomputer. This uses the scp command again, but this time with the -r (recursive) option, which will copy the directory and all subdirectories and files within it. On the local computer enter the following commands: cd ~ scp r seuss/ <username>@tango.vpac.org: Now on the supercomputer do a directory listing but specify the file you want and with the time option. You should see a list.txt in your home directory (the original one) and a seuss directory. Move into the seuss directory and run a directory listing again with the same option. There should be another list.txt file, the one you just moved, but you will notice it has a different timestamp. ls l list.txt cd seuss ls l list.txt Sometimes you may wish to compare the content of files as well as when they were created. To do this use the diff command. This compares files lineby-line and prints the differences to the screen. As usual there are a number of options which can be ascertained from the command man diff, but for now we'll just use the basic command. The command uses brackets to indicate where additional material is located. To illustrate this, let's add some lines to the first list.txt file and then run the diff comparison. cd ~ nano list.txt 33

34 Add the following lines. Yes, some are red. And some are blue. Some are old. Some are new. Write out (Cntrl-O) and exit (Cntl-X) and run the diff command. diff list.txt seuss/list.txt The output should be something like the following: < Yes, some are red. And some are blue. < Some are old. Some are new. > This one has a little star. > This one has a little car. > Say! What a lot of fish there are. For a side-by-side representation use the command sdiff instead. 8. Searching for and within files and Wildcards Often you will want to search for files or search within files for a particular phrase. The find command, which will find files according to the directory and subdirectories offered, by name, modification time, size etc, and with filter operations, all of which are available through man find. To find all files with the suffix.txt on your supercomputer account use the following command: cd ~ find. name '*.txt' Note that the filter is within quotes, to ensure that the command is not expanded due to the wildcard. To search within a collection of files use the grep command. It originally an abbreviation of "global search for a regular expression and print to standard output". The command searches the named input files for lines containing a match to the given pattern, including regular expressions, and prints the matching lines. As usual there are a variety of options available through man grep. The following command will search for the pattern 'red', ignoring case, within the directory seuss. Enter the following on the supercomputer: cd ~ grep i red seuss/* Note that the files being searched must be specified, even with a wildcard character. Simply stating a directory is insufficient. Where there are 34

35 multiple results, grep will also display the filename. Compressed or gzipped files can be searched with zgrep. The wildcard you see most often is * (asterisk), but we'll start with something simpler:? (question mark). When it appears in a filename, the? matches any single character. For example, letter? refers to any filename that begins with letter and has one character after that. This would include lettera, letter1, as well as filenames with a non-printing character as their last letter, like letter^c. The * wildcard matches any character or group of zero or more characters. For example, *.txt matches all files whose names end with.txt, *.c matches all files whose names end with.c (by convention, source code for programs in the C language), and so on.xactly how is this different to what Telecom was in the 1980s? It might be a well-engineered network, but it will still be too dear in the city. It will still have farmers arguing; I decided to live 50kms out of town, but I don t see why the government shouldn t give me everything a townie has. It will still have union, green, depressed-area and retail-pressure groups trying to manipulate the politicians and the press. Wildcard Matches? Any single character * Any group of zero or more characters [ab] Either a or b [a-z] Any character between a and z, inclusive 9. Delete files and directories Sometimes you'll want to remove files and directories from your account. Be very careful and very selective with this because when you're operating on the command line there's no "trashcan" to easily undelete files. Somewhere, delete really means what it says, and that somewhere is here. On the supercomputer we'll carefully delete the file in the home directory and then change the directory to seuss and delete the file there. We'll delete the file in that directory, change out of that directory and delete the directory. cd ~ rm list.txt cd seuss rm list.txt cd ~ rmdir seuss 35

36 Then on the local computer we'll use a shortcut; a command which deletes the entire directory, all subdirectories and all files within the directory tree. This is remove with the recursive and force options. cd ~ rm rf seuss Be very careful with rm, especially with the -rf option and especially with wildcards. Consider what would happen to someone who wishes to delete all their backup files in a directory with the helpful suffix.bak. Choosing a wildcard and the suffix they intend to type rm *.BAK but instead, they mistype the command and type rm *.BAK. The result of this typing error is that they have just deleted everything in that directory. Worse still imagine a user running as root thinking that they are about to delete a directory and instead types rm rf / ; a command that will delete everything or, more commonly rm rf./ ; a command which deletes the current directory and all subdirectories. 36

37 3.0 Environment Modules and the Portable Batch System Environment Modules Environment modules (not to be confused with kernel modules - another topic for another day) provide for the dynamic modification of the user's environment via module files. Each module contains the necessary configuration information for the user's session to operate according according to the modules loaded, such as the location of the application, its manual path, LD_LIBRARY_PATH and so forth - it is a lot easier that having to set these every time an application is used! Modulefiles also have the advantages of being shared on many users on a system (such as an HPC system) and easily allowing multiple installations of the same application but with different versions and compilation options. Module commands Some basic module commands include the following: 1. Listing Available Modules module avail This option lists all the modules which are available to be loaded. Notice that many of them have version numbers associated with them. Modules makes it easy to switch compiler application versions. The module name without a version number is the production default. 2. Module Specific Help module help [modulefile] If a module looks interesting, to get more information use this command to display the `help' information contained within the given module file. Note that this will only work if the module file has help associated with it. For example, module help namd will provide no additional information. 3. Displaying a Module's Description 37

38 module whatis <modulefile> This option provides a description of the module listed. 4. Displaying a Module's Variables module display <modulefile> Use this command to see exactly what a given modulefile will do to your environment, such as what will be added to the PATH, MANPATH, etc. environment variables. For example the command: module display freesurfer Will display the following results: /usr/local/modules/modulefiles/freesurfer/4.5.0: module whatis Set environment variables to use freesurfer version prepend path PATH /usr/local/freesurfer/4.5.0/bin prepend path FREESURFER_HOME /usr/local/freesurfer/4.5.0 prepend path SUBJECTS_DIR /usr/local/freesurfer/4.5.0/subjects 5. Loading A Module This adds one or more modulefiles to the user's current environment (some modulefiles load other modulefiles, e.g., module load vpac or module load gcc). You can load multiple modules simultaneously. module load <modulefile> For example, to load NAMD one would use the command: module load namd If you load the generic name of a module, you will get the default version. To load a specific version, load the module using its full specification. 38

39 for example; module load gcc/ Listing Loaded Modules module list This lists all the modules which are currently loaded into the users environment. 7. Unloading Modules module unload <modulefile> This removes any listed modules from the user's current environment. Multiple modules can be unloaded simulatenously e.g., module unload gcc namd Note that unloading a module does not remove additional modules loaded by the original module file. For example; module load gcc module list Currently Loaded Modulefiles: 1) gmp/ ) mpfr/ ) gcc/4.3.4 module unload gcc module list Currently Loaded Modulefiles: 1) gmp/ ) mpfr/ Switching Modules module switch <modulefile1> <modulefile2> This unloads one modulefile 39 (modulefile1) and loads another

40 (modulefile2). This is very handy for testing behavior across different versions of the same program. For example, the following loads Octave v3.2.0, but the user then discovers that an earlier version better suits their needs. Rather than loading and unloading the different versions they can simply use module switch instead. module load octave/3.2.0 module switch octave/3.2.0 octave/ Unloading All Modules module purge This removes all modules from the user's environment. 10. Automatic Module Loading, Unloading and Switching The 'module initadd' command can be used to have a module automatically loaded each time a user logs into the system; module initadd <modulename> Depending on the user's shell preferred shell, the initadd command searches a range of initialisation files in the user's home directory for a 'module load' and appends as appropriate. For the VPAC default of bash, the order is.modules,.bash_profile,.bash_logion If the module load line is found in multiple shell initialization files, all of the lines are changed. To remove a module file from the initialisation files use; initrm <modulefile> To switch one modulefile to another in the initialisation file use; initswitch <modulefile1> <modulefile2 > To list all the modulefiles loaded from the shell initialisation files use; initlist To clear all the modulefiles from the shell's initialisation files use; initclear 40

41 Portable Batch System The Portable Batch System (or simply PBS) is the name of a utility software that performs job scheduling by assigning unattended background tasks expressed as batch jobs, among the available resources. The schedular provides for paramterisation of computer resources, an automatic submission of execution tasks, and a notification system for incidents. There is a variety of PBS applications available. At VPAC we use TORQUE, derived from OpenPBS, which is actively developed, supported and maintained by Cluster Resources Inc. Submitting and running jobs is a relatively straight-forward process consisting of: 1) Setup and launch 2) Monitor results 3) Retrieve results and analyse. 1. Setup and Launch When uploading files in preparation for launching a HPC job it is good practice to keep each job in separate folders, labeled in an intuitive way, such as: NAMD_albumin_run_01. Writing all files to the top level home directory will very quickly become difficult to follow and easy to make mistakes! Use the mkdir command learned in the previous section to do this. A pbs script allocates the compute nodes and time requested by the user The user can edit this script to change the number of preferences including the number of cpus to use, the length of time to run the job and the name of the program executable. An example PBS script can be copied into the training account home directory on the supercomputer from the NAMD tutorial that follows; cd ~ cp /common/examples/namd_training_example/pbs_example_script. Lines that start with #PBS space does not. pass a PBS command, while adding a white For example, compare the lines: #PBS l walltime=24:0:0 <- works! # PBS l walltime=24:0:0 <- doesn't work. 41

42 Example pbs_batch_script: less pbs_example_script #!/bin/bash Explanation: <-At the start of all scripts. # pbs script for a NAMD job: <- A comment line. #PBS N Job_name <- The name of the job. #PBS l nodes=8 <- Asking for 8 cores. #PBS m ae <- Mail user if job aborts (a) or ends (e) (currently commented out). #PBS l walltime=24:10:5 <- Set the job to run 24 hours 10 minutes and 5 seconds. #PBS o output_file_name.txt <-Set output file name. module load namd <- Load the NAMD module cd $PBS_O_WORKDIR <- Change to the Work directory. # Launching the job! <- Another comment line. mpiexec namd2 config_file <- Launching the executable namd2 with the file config_file. Nodes: Requesting more nodes for a particular job does not necessarily mean better performance or faster completion. This depends on how well parallelized the program is. Also, requesting large numbers of cpus (> 64) may result in the job waiting in the queue for days or weeks while the scheduler makes resources available. For example, a 32 cpu job running in 5 days, may take 64 cpus 7 days (3 days to run plus and extra 4 days while waiting in the queue!). Walltime: Walltime is the length of time specified in the pbs script for which the job will run. Make sure that you have specified enough walltime for a particular job to complete! Your job will end after the allocated walltime whether it is finished or not, sometimes resulting in lost data if the program does not check point restart files. If you are not sure how long a job will run, set a generous walltime, and then check the job status before the walltime expires. If your job needs more time simply help@vpac.org and ask the system administration to change the walltime for the job. 42

43 Once you are satisfied with your pbs_batch_script and have uploaded all the input files and necessary modules for your job simply type at the command line (Don't do this yet - it is an example!): qsub pbs_example_script 2. Monitoring Progess Once a job is launched it is possible to monitor how it is progressing. One way to do this is type at the command line: showq This will show all current jobs on the cluster so to pick out the lines relevant to you use the ever-useful grep command: showq grep <username> If all is well you should see a line like: > showq grep joe joe Running 16 23:26:49 Wed Feb 13 14:38:33 Reading across the line, the first number is the job number, (83238), owned by joe and the status of the job is Running. The job is running on 16 cpus and has 23 hours 26 minutes and 49 seconds remaining. It was submitted on Wed Feb 13 at 14:38:33. Another way to see this information is with the -u option: showq u <username> If the cluster is busy, you job may have to wait in the queue, in which case the status of the job would be Idle. If your job is listed as Idle, one can check when it is likely to start by the command: showstart <job_number> If you don't see any line then either your job has finished or died! If you know the job number you can view the output while it is still running by typing at the command line: jobstatus o <job_number> Likewise, to see error messages, (if any), type jobstatus e <job_number> 43

44 If wish to stop your job prematurely, you may do so with the command: qdel <job_number> 3. Retrieving and Analysing Results Often it is most convenient, depending on the type of simulation, to download the output of job to your local computer for analysis, using the scp methods explained in chapter 2. If you have run your job in a folder then it is simply a matter of downloading that entire folder to your desktop. Most post-processing the data can usually be done happily on most desktops and don't require further HPC resources at that point. Frequently Used PBS Commands The following is a list of frequently used PBS commands. qstat less Information about queues and jobs, piped through less. qstat q List all queues qstat Q Queue limits. qstat a List all jobs in alternative format. qstat au List all jobs of a particular user, e.g., qstat -au mike qstat s less All jobs with status comments, piped through less qstat r less All running jobs, piped through less qstat f <jobid> Information concerning a particular job. qsub <pbsscript> Submits pbscript qsub I Submit an interactive batch job. qdel <jobid> Deletes a job 44

45 Grid Australia The grid is an emerging technology in high performance computing whereby a grid framework is put in place to allow trusted, certified users to submit jobs on remote clusters. The grid software decides where the job should run depending on requirements thus making best use of computational resources and allowing high capacity users access to greater processing power. GridAustralia, managed by ARCS (Australian Research Collaboration Service) contributes to a range of grid middleware products developed to meet local requirements. These include; Globus An open source toolkit designed for building computer grids Myproxy, OpenCA, VOMRS Components that add to the basic globus structure. Grix A communications tool that connects from the user's Desktop to OpenCA to assist in x509 certification management. Grisu A lightweight application library with a simple GUI existing between the user's Desktop and the Grid middleware. If you require large computational resources, you will be interested in using the grid. help@arcs.org.au for more details. 45

46 4.0 High Performance Computing Tutorials The aim of these tutorials is to give the user experience to upload and submit a typical parallelised job to the cluster, download the output and visualize the preliminary results, where applicable. Examples are provided for NAMD (life sciences), Abaqus (mechanical engineering) and Octave (numerical computations). In each, the workflow is representative of launchin other parallel jobs. NAMD and VMD NAMD is a parallel, molecular dynamics simulation program used to model large biomolecular systems using high performance computing clusters which is freely available for academic work. If you are interested in running NAMD simulations you should also install a local copy of VMD on your own computer. VMD is a molecular viewer program that can be used to help set up NAMD simulation and to help analyse and visualize NAMD output. VMD can be freely obtained from: Additional tutorials and information about NAMD are available at: Molecular Dynamics Simulation We have an example of Ubiquitin protein, in a vacuum, ready to download and run. (Normally we would do this with water and salt ions, but we need the simulation to run quickly for demonstration purposes!), After logging on to one of the clusters, copy the entire example directory to your home directory. cp r /common/examples/namd_training_example. Change into this directory and launch the job with the command "qsub" and the pbs script. cd NAMD_training_example qsub pbs_example_script Check the job is running with the showq command. 46

47 Too much information? Try: showq grep <your account name> This particular job is very short compared to a regular NAMD simulation and should be finished in about 2 minutes. As the job runs, various output files are produced but the main one you will be interested in is the trajectory file with the.dcd suffix. While we wait, let's take a look at the pbs example script to understand what is going on. Type: less pbs_example_script See the lines: # how many cpus? #PBS l nodes=4 This tells us how many CPUs we are going to use. (in this case, 4) When you run bigger jobs you'll want to use more CPUs to speed up the calculations! Another important line is the walltime: # How long to run the job? (hours:minutes:seconds) #PBS l walltime=0:15:0 This tells us how long to run the simulation (in this example, 15 minutes). The job will end when this time expires, whether the job has finished or not! Make sure to have a good estimate of walltime for your real jobs! Typically you might run a NAMD job for days, if not weeks. The business end of the script is in the last line: mpiexec namd2 Ubiquitin_example.conf This is saying use the program "mpiexec" to launch the parallel program "namd2" passing the configuration file "Ubiquitin_example.conf" So far so good! Press "q" to quit viewing the pbs script. Now lets have a look at the NAMD configuration script: less Ubiquitin_example.conf 47

48 There is quite a bit of information here, don't worry though. Most of the parameters stay the same for each simulation. Let me just point out the important bits. See the lines near the top: structure coordinates outputname 1ubq_example.psf 1ubq_example.pdb 1ubq_example_output_01 These are simply defining the input files, (the protein structure file.psf, and the coordinate file,.pdb) and also the name of the output files. Further down you will see: set temp 310 temperature $temp Which is setting the temperature to 310 K ( 37 C) while below that we have: ## Parameter file: paratypecharmm on parameters par_all27_prot_na.inp This tells NAMD which parameter file to use. (you'll see a copy of the parameter file in the NAMD_example directory). Depending on what molecules you have in you simulation, you may need to use a more detailed parameter file. Somewhere in the middle you will see these lines: ## Periodic Boundary Conditions cellbasisvector cellbasisvector cellbasisvector cellorigin wrapall on wrapwater on This defines the boundary conditions, ( a cheats way to simulate an infinite system), where a molecule coming off one side of the boundary will reappear on the other. Near the bottom we have the lines: ## Output files: restartfreq

49 dcdfreq 50 xstfreq outputenergies outputpressure These lines tell us how often to write out to the output files. The most important is the dcdfreq, (here set to 50), or how often to save a frame of the dynamics trajectory. (Usually this is set around 5,000 to 10,000). The dcd output file can become ridiculously HUGE if this is too small. 50 is OK for this short example! The last few line in the configuration file: ## Minimize, reinitialize velocities, run dynamics: minimize 100 reinitvels $temp run 5000 These tell us that the simulation is first minimized for 100 steps, (a good idea to iron out bad contacts and steric clashes, but usually set to 1000 steps or more), followed by resetting the velocities to $temp, (310 K, remember?), and then told to run only a mere 1000 steps. (This is a very short example!). Typically you might set "run" to or more. Press "q" to quit viewing the configuration file. Check again on the status of your job: showq u <your account name> If you don't see anything it probably means the job has finished. List your directory and you should see something like: ls lrt rw r r 1 joe vpac Feb 20 12:54 par_all27_prot_na.inp rw r r 1 joe vpac Feb 20 12:54 1ubq_example.pdb rw r r 1 joe vpac Feb 20 12:54 1ubq_example.psf rw r r 1 joe vpac 471 Feb 20 12:58 pbs_example_script rw r r 1 joe vpac 2614 Feb 20 13:02 Ubiquitin_example.conf rw 1 joe vpac 0 Feb 20 13:02 namd_example_job_01.e rw 1 joe vpac 166 Feb 20 13:03 1ubq_example_output_01.xst rw 1 joe vpac Feb 20 13:03 Namd_1ubq_example_output.txt rw 1 joe vpac 224 Feb 20 13:03 1ubq_example_output_01.xsc 49

50 rw 1 joe vpac Feb 20 13:03 1ubq_example_output_01.coor rw 1 joe vpac Feb 20 13:03 1ubq_example_output_01.vel rw 1 joe vpac Feb 20 13:03 1ubq_example_output_01.dcd The highlighted file is the main output file while the.xsc,.coor,.vel files all have to do with being able to restart the simulation at a later date, while the Namd_1ubq_example_output.txt file contains the text output from the simulation. Congratulations! You have just run a short molecular dynamics simulation on the cluster. Next, we'll copy that information back to your local computer and use VMD to visualize the results. Visualizing NAMD results with VMD In part 2 of the NAMD tutorial, where you will be using the molecular visualization program, VMD to look at the trajectory data of the ubiquitin protein you generated in part I of the tutorial. If you haven't already done so, download and install VMD to your local computer (make sure to pick the right flavour, Windows, Mac, or Linux). Download the entire NAMD example directory back to your local computer. For example, in Linux you can use: scp r <your account name>@tango.vpac.org:namd_training_example. You can now start VMD and load up the trajectory data. In a new terminal type: vmd Three windows should pop up: The Main panel 50

51 The Display The Console The first file you need to read into VMD is the protein structure file, (1ubq_example.psf in this case). The.psf file contains important information about the system such as which atoms are bonded together, what charge they are, and the mass of each atom type, but does not contain any spacial information. From the Main panel: File > New molecule > (browse: 1ubq_example.psf) > load You should see nothing in the display, but an entry in the Main panel. Next load the coordinates from the pdb file: File > Load data onto molecule > (browse: 1ubq_example.pdb) > load Now you should have the model in the display that can be moved around with the mouse. This is the initial starting position of the simulation. Next load in the trajectory data into VMD: File > Load data onto molecule > (browse: 1ubq_example_output_01.dcd) > load 51

52 This data is shows how the atoms in the model moved around during the course of the NAMD simulation. You can play the trajectory by pressing the bottom right arrow of the Main panel. (Use the speed scroll bar to the left of that button to slow it down). What you are seeing is supposed to represent is the molecular motion of the protein on an extremely small time scale. The NAMD program calculates how the molecule moves over time given certain parameters such as temperature. These models and simulations can give insight into how proteins behave and their role in biological function and certain diseases. From the main panel you can bring up the Representations dialog to play with more rendering types: - have a play! Graphics > Representations Using Different Compilers and Processors Different compilers and quantities of processors will finish the tasks at different speeds. Knowing what sort of speeds you can achieve is handy for estimating walltime. To modify the NAMD job you will need to alter the module and the number of processors in the pbs script (pbs_example_script). Try switching between the GCC compiled version of OpenMPI with the PGI compiled version to compare the speed differences of our MPI code: GNU GCC PGI Intel 1 CPU 2 CPU 4 CPU Abaqus: Finite Element Analysis This tutorial demonstrates how to run and view the results of an explicit crash simulation using the ABAQUS suite of FEA products on VPAC's HPC facility. Please not that model pre-processing is not discussed. 52

53 The case study is a car door being propelled into a pole. This is analogous to the EURONCAP pole test, in which a car is propelled sideways into a rigid pole of diameter 254 mm at 29 km/h. While a crash event generally lasts for around 100 milliseconds, the time step of the case study has been reduced to 10 milliseconds to reduce the job time. A schematic of the assembly is shown in the following figure. Text inside [Square Brackets] indicates that there is a button to push in the GUI of the program being described. Menu command sequences are linked by ">" symbols. 2. Nomenclature CAE Complete ABAQUS Environment EURONCAP European New Car Assessment Programme ODB Output Database 3. Job Submission Serious compute loads must be launched via PBS on Tango. Copy the job files to your working directory using the following commands: ssh tango.vpac.org cp r /common/examples/abaqus/door. cd door/ ls The files copied to your working directory are shown below: Door.cae - ABAQUS model database file Door.jnl - ABAQUS model database journal file 53

54 Door.inp - ABAQUS input deck Door.odb - ABAQUS output database file pbs-abaqus - pbs job submission script A sample PBS script for ABAQUS is shown below. #!/bin/bash #PBS l nodes=1:ppn=1 #PBS l walltime=00:05:00 # Go to the directory from which you submitted the job cd $PBS_O_WORKDIR module load abaqus # # Run the job 'Door' abaqus job=door The script above runs on a single node. The following runs on 12 but is using HP-MPI running on Gig-Ethernet; it cannot run on Infiniband. #!/bin/bash #PBS l nodes=12 #PBS l walltime=0:05:00 #PBS N AbaqusDoor # Go to the directory from which you submitted the job cd $PBS_O_WORKDIR module load abaqus cpus=$(wc l $PBS_NODEFILE awk '{print $1}') mp_host_list="[" for n in $(sort u $PBS_NODEFILE) do mp_host_list="${mp_host_list}['$n',$(grep c $n $PBS_NODEFILE)]," done mp_host_list=$(echo ${mp_host_list} sed e "s/,$/]/") echo "mp_host_list=${mp_host_list}" > abaqus_v6.env # # Run the job 'Door' abaqus mp_mode=mpi cpus=$cpus job=door Submit the job using the following command: qsub pbs abaqus The status of the job can be queried using the following command: tail f door.sta 4. Post-Processing Once the job has completed, all files, with the exception of the output database (.ODB) file can be deleted. By default, ABAQUS/CAE writes the results of the analysis to the ODB file. When you create a step, ABAQUS/CAE 54

55 generates a default output request for the step, which in the case of this analysis is Energy Output. You use the Field Output Requests Manager to request output of variables that should be written at relatively low frequencies to the output database from the entire model or from a large portion of the model. You use the History Output Requests Manager to request output of variables that should be written to the output database at a high frequency from a small portion of the model; for example, the displacement of a single node. The results will be visualised using ABAQUS/CAE. It should be noted that ABAQUS/Viewer is a subset of ABAQUS/CAE that contains only the postprocessing capabilities of the Visualization module. The procedure discussed in this tutorial also applies to ABAQUS/Viewer. The procedure used to load ABAQUS/CAE on Tango is shown below. module load abaqus abaqus cae The following procedure is used to open the ODB file; File 3. Click [Open Database] in the Session Start window. The Open Database dialog will appear. Select Output Database from the Filter dropdown menu. Select Door.odb and click [OK]. By default, ABAQUS/CAE will plot the undeformed shape with exterior edges visible. For clarity (if the mesh density is high) it may be necessary to make feature edges visible. The following procedure is used: 1. Click [Common Plot Options] in the Toolbox Area. 2. In the Basic Tab, check Feature edges in the Visible Edges section. 3. Click [OK]. The door assembly undeformed shape plot is shown in the following figure. Both exterior edges and feature edges are shown. 55

56 4.1. Deformed Shapes The following procedure can be used to plot the crash models deformed shape: 1. Click [Plot Deformed Shape] in the Toolbox area. By default, the final step is displayed. It should be noted that the Deformation Scale Factor is 1 by default in explicit analyses. 2. Click [Animate: Time History] to animate the crash event. The frame rate can be adjusted by clicking [Animation Options] and moving the slider in the Player tab to the desired speed. The door assembly deformed shape plot is shown in its final increment in the following figure. The following procedure can be used to plot contours over the deformed shape: 1. Click [Plot Contours on Deformed Shape] in the Toolbox area. 2. If the user intends to compare a number of different configurations of the same model, the minimum stress/strain value should be set to the same value for each configuration. Click [Contour Plot Options], select the limits tab and check Specify and set the value to 0. In analyses were non-linear geometry is considered, the user can also specify a maximum strain/stress to help gauge the location(s) of any plastic regions. 3. The output variable can be modified by clicking Result > Field Output. By default Mises stresses are plotted. A number of invariants are available for each output. Note that the field output variable can also be modified in the Probe Values dialog. The door assembly contour on deformed shape plot is shown in the following figure. 56

57 The stresses and strains of individual elements can be probed as follows: 1. Click Tools > Query and select Probe Values. Click [Apply]. The Probe Values dialog will appear. 2. Check the appropriate output variable in the Probe Value section and click on an element of choice. The probed values of each element selected can then be written to a text file. It should be noted that default probe location is the integration point of the element. Hence, for this example, one value will be stored to the Selected Probe Values table since the element is a reduced integration shell of type S4R. It should also be noted that the stresses and strains shown in the contour plot are an average of the actual stress that is calculated at the integration point. The following procedure can be used to save image animation: 1. Plot a deformed shape or contour on deformed shape using the procedure described above. 2. Animate the results event using the procured described above. 3. Click Animate > Save As. Accept the default file format (AVI) if you plan to use the animation in a PowerPoint presentation. However, it should be noted that QuickTime format offers better compression and image quality. 4. Specify a file name and location and click [OK]. 5. It should be noted that viewport annotations can be toggled off for clarity. Click Viewport > Viewport Annotation Options and uncheck any information that is not required XY Data Plotting History output can be used to plot XY Data. In this example, all energy output variables (whole model) were written to the ODB file. The user can save a numerous output variables for node/element sets that are present in a model. The following procedure can be used to plot XY Data: 1. Click [Create XY Data], accept the default selection (ODB history output) and click 57

58 [Continue]. 2. Select Internal Energy and Kinetic Energy and click [Plot]. All XY data can be exported to Excel in.csv format. A sample XY plot is shown in the following figure showing kinetic energy and internal energy versus time. It should be noted that it may be necessary to "operate" on XY Data to create certain plots that are not time dependent (Force-displacement etc). The following procedure will demonstrate this process: 1. Click [Create XY Data], accept the default selection (ODB history output) and click [Continue]. 2. Select data set 1 and click [Save As]. Give the data an appropriate name. 3. Repeat step 2 for data set 2. Click [Dismiss]. 4. Click [Create XY Data] and select Operate on XY Data. 5. From the Operators section, select combine(x,x). Double click the x-axis data, and then double click the y-axis data. Click [Save As] followed by [Plot Expression]. R Statistics and Grisu Tutorial 1. Establishing a Grisu Account R tutorial derive from from the material provided by Clarkson University, Department of Mathematics. R is a programming language for statistical computing and graphics. In this tutorial we are going to generate some basic plots using R on an HPC system, but also using the Grisu application with he ARCS Compute Cloud. This provides a graphic front-end for the user to submit HPC jobs. Users must have Java 1.5 (or a newer version) installed to run applications on the ARCS Compute Cloud. Also, they must register as an ARCS 58

59 Compute Service user. After registering, users can log on to the Compute Cloud with their institutional user name and password and authenticate using the IDP (e.g., at Your ARCS Compute Cloud can be mapped to an existing HPC account. You need only do this once, but you must individually map each existing HPC account that you have. 2. Download the Files and Applications Once you have an ARCS cloud account and Java installed you'll need the appropriate Java webstart aplication. For this tutorial we will need grisu-r.jnlp available from Secondly, we'll need two data files, trees91.csv and w1.dat and a R script files. These are all located in the /common/examples/r directory. Copy both these files from supercomputer to the Desktop machine. So, from the Desktop machine; cd ~ scp trainxx@tango.vpac.org:/common/examples/r/trees91.csv. scp trainxx@tango.vpac.org:/common/examples/r/w1.dat. scp trainxx@tango.vpac.org:/common/examples/r/tutorial.r. The data is used to indicate an estimate of biomass of ponderosa pine in a study performed by Dale W. Johnson, J. Timothy Ball, and Roger F. Walker who are associated with the Biological Sciences Center, Desert Research Institute, University of Nevada. It consists of observation measurements and markers for 28 different measurements of a given tree. For example, the first number in each row is a number, either 1, 2, 3, or 4, which signifies a different level of exposure to carbon dioxide. The sixth number in every row is an estimate of the biomass of the stems of a tree. Note that the very first line in the file is a list of labels used for the different columns of data. 3. The R Script A review of the R script tells us what we're doing. Firstly, we import the w1.dat and trees91.csv files into appropriate variables. Then we plot a histogram, the breaks, the size of the domain, titles and a strip chart on to the histogram. Each of these actions is illustrated. # Import the tree data CSV w1 < read.csv(file="w1.dat",sep=",",head=true) tree < read.csv(file="trees91.csv",sep=",",head=true) # Plot a histogram of the data hist(w1$vals) # Specify the number of breaks to use hist(w1$vals,breaks=2) 59

60 hist(w1$vals,breaks=4) hist(w1$vals,breaks=6) hist(w1$vals,breaks=8) hist(w1$vals,breaks=12) # Specify the size of the domain using xlim hist(w1$vals,breaks=12,xlim=c(0,10)) hist(w1$vals,breaks=12,xlim=c( 1,2)) hist(w1$vals,breaks=12,xlim=c(0,2)) hist(w1$vals,breaks=12,xlim=c(1,1.3)) hist(w1$vals,breaks=12,xlim=c(0.9,1.3)) # adding titles and labels always annotate your plots! hist(w1$vals,main='leaf BioMass in High CO2 Environment',xlab='BioMass of Leaves') title('leaf BioMass in High CO2 Environment',xlab='BioMass of Leaves') # add other kinds of plots (e.g., stripchart) hist(w1$vals,main='leaf BioMass in High CO2 Environment',xlab='BioMass of Leaves',ylim=c(0,16)) stripchart(w1$vals,add=true,at=15.5) 4. Submitting the Job via Grisu The next step it to start up Grisu, and select the 'Job Submission' tab and the 'General' subtab. Give the job a name (R_job), set a walltime (15 minutes), and the number of CPUs (1) to use - rather like a PBS script! Then select a location where the job is being submitted; we've chosen Canterbury University in New Zealand in the example below. Then, under the 'Job Parameters' subtab load the necessary files; browse for the tutorial.r file from the 'R Script' section, and add the w1.dat and trees91.csv under the 'other input files' section. Then go to the 'File Management' tab and check that they have been uploaded to the HPC system. If that is the case go back to the 'Job Submission' tab and select the 'Submit' button on the bottom right. Your job is now on its way! Within 15 minutes (probably less) it will be finished. Go back to the 'File Management' page and refresh. You should see a Rplots.pdf file. Using the interface provided, copy this 60