Mathematical Libraries and Application Software on JUROPA and JUQUEEN

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Mathematical Libraries and Application Software on JUROPA and JUQUEEN JSC Training Course May 2014 I.Gutheil

2 Outline General Informations Sequential Libraries Parallel Libraries and Application Systems: Threaded Libraries MPI parallel Libraries Application Software Further Information May 2014 I.Gutheil Folie 2

3 General Informations JUROPA (I) Three major Compiler versions Default: Intel with MKL Module: Intel with MKL Module: Intel with MKL Module: Intel 12.0.[3, 4, or 5] with MKL included Module: Intel 12.1.[0, 1, 2, or 4] with MKL included Module: Intel or , MKL included module unload intel, module unload mkl before loading a new intel (and MKL) Module module switch parastation parastation/mpi2-intel for some new libraries necessary May 2014 I.Gutheil Folie 3

4 General Informations JUROPA (II) Most libraries compiled with and MKL , new versions with 12.0, 12.1, or 13.1 only Module unload and module load must be called in batch scripts before execution, too Starting with intel/ MKL is included with module load intel/12.* For most libraries versions compiled with intel/12.0.* and/or 12.1.* available Latest libraries compiled with intel/ May 2014 I.Gutheil Folie 4

5 General Informations JUROPA (III) module avail lists names of available libraries module help name shows how to use library module load name prepends LD LIBRARY PATH, LIBRARY PATH and INCLUDE and sets NAME ROOT to the correct directory Link sequence important,.o always before the libraries, sometimes double linking necessary May 2014 I.Gutheil Folie 5

6 General Informations JUQUEEN (I) All libraries as modules in /bgsys/local/name module avail lists names of available libraries module help name tells how to use library module load name sets environment variables for -L$(* LIB) and -I$(* INCLUDE) to include in makefile Link sequence important,.o always before the libraries, sometimes double linking necessary May 2014 I.Gutheil Folie 6

7 General Informations (II) First all libraries will be compiled with -O3 -qstrict -g qsimd=noauto Additional version compiled without -g added also some versions with simd See module avail for available versions May 2014 I.Gutheil Folie 7

8 Sequential Libraries and Packages (I) Vendor specific Libraries JUROPA only MKL Intel R Math Kernel Library versions as mentioned in general informations JUQUEEN only ESSL (Engineering and Scientific Subroutine Library) version 5.1 in /bgsys/local/lib May 2014 I.Gutheil Folie 8

9 Sequential Libraries and Packages (II) Public domain Libraries LAPACK (Linear Algebra PACKage) ARPACK (Arnoldi PACKage) planned GSL (Gnu Scientific Library) GMP (Gnu Multiple Precision Arithmetic Library) May 2014 I.Gutheil Folie 9

10 Contents of Intel R MKL 10.* BLAS, Sparse BLAS, CBLAS LAPACK Iterative Sparse Solvers, Trust Region Solver Vector Math Library Vector Statistical Library Fourier Transform Functions Trigonometric Transform Functions May 2014 I.Gutheil Folie 10

11 Contents of Intel R MKL 10.* GMP routines Poisson Library Interface for fftw For more information see http: // Software/SystemDependentLibraries/MKLJuropa.html May 2014 I.Gutheil Folie 11

12 Contents of ESSL Version 5.1 BLAS level 1-3 and additional vector, matrix-vector, and matrix-matrix operations Sparse vector and matrix operations LAPACK computational routines for linear equation systems and eigensystems Banded linear system solvers Linear Least Squares Fast Fourier Transforms May 2014 I.Gutheil Folie 12

13 Contents of ESSL Version 5.1 (II) Numerical Quadrature Random Number Generation Interpolation For further information see IBM Engineering and Scientific Subroutine Library for Linux on POWER V5.1: Guide and Reference http: // Software/SystemDependentLibraries/ESSL_ESSLSMP.html Link to IBM documents Guide and Reference May 2014 I.Gutheil Folie 13

14 Usage of MKL (I) FORTRAN, C, and C++ callable Arrays FORTRAN like, i.e. column-first (except cblas) Three versions, old and two variants of 10.2 starting with intel/ included in intel module Compilation and linking of program name.f calling sequential MKL routines, default version: ifort name.f -o name -lmkl intel lp64 -lmkl intel thread -lmkl core -liomp5 -lpthread or ifort name.f -o name -lmkl intel lp64 -lmkl sequential -lmkl core -liomp5 -lpthread May 2014 I.Gutheil Folie 14

15 Usage of MKL (II) Compilation and linking of program name.f calling sequential MKL routines starting with intel/ module unload mkl module switch intel intel/ ifort name.f -o name -lmkl intel lp64 -lmkl sequential -lmkl core -liomp5 -lpthread Linking of MKL always dynamic, so modules must be switched before execution, too May 2014 I.Gutheil Folie 15

16 Usage of MKL (III) To use CBLAS include mkl.h into source code Compilation and linking of program name.c calling sequential MKL routines [module unload mkl module switch intel intel/12.0.3] icc name.c -o name -lmkl intel lp64 -lmkl sequential -lmkl core -liomp5 -lpthread [-lifcore -lifport] May 2014 I.Gutheil Folie 16

17 Usage of ESSL FORTRAN, C, and C++ callable, Arrays FORTRAN like, i.e. column-first Header file essl.h for C and C++ Installed in /bgsys/local/lib (not as module) May 2014 I.Gutheil Folie 17

18 Usage of ESSL (II) Compilation and linking of program name.f calling ESSL routines mpixlf90 r name.f -L/bgsys/local/lib -lesslbg Compilation and linking of program name.c calling ESSL routines mpixlc r name.c -I/opt/ibmmath/essl/5.1/include -L/bgsys/local/lib -lesslbg -L/opt/ibmcmp/xlf/bg/14.1/lib64 -lxl -lxlopt -lxlf90 r -lxlfmath -lm -lrt May 2014 I.Gutheil Folie 18

19 LAPACK (I) Part of MKL on Juropa, until intel/11.1.* seperate file, starting with intel/ in libmkl core.a Public domain version 3.3 and on JUQUEEN Experimental C-LAPACK, liblapacke.a in version on JUQUEEN Must be used together with ESSL (or ESSLsmp) Some routines already in ESSL Attention, some calling sequences are different! Experimental LAPACK header file available for C-usage of lapack 3.3 on JUQUEEN (may also be tried with 3.4.2) May 2014 I.Gutheil Folie 19

20 LAPACK (II) Compilation and linking of FORTRAN program name.f calling LAPACK routines JUROPA: (see usage of MKL), -lmkl lapack only up to intel/ JUQUEEN: module load lapack/3.4.2[ g][ simd] mpixlf90 r name.f -Wl,-allow-multiple-definition -L/bgsys/local/lib [-lessl[smp]bg] -L$(LAPACK LIB) -llapack -lessl[smp]bg ESSL must be linked after LAPACK to resolve references May 2014 I.Gutheil Folie 20

21 Arpack ARPACK, ARnoldi PACKage, Version 2.1 Iterative solver for sparse eigenvalue problems Reverse communication interface FORTRAN 77 Calls LAPACK and BLAS routines May 2014 I.Gutheil Folie 21

22 GSL GNU Scientific Library Version 1.13, 1.14(default), and 1.15 with intel/ on JUROPA, 1.15 on JUQUEEN Provides a wide range of mathematical routines Not recommended for performance reasons Often used by configure scripts module load gsl[/1.13][/1.15] JUROPA module load gsl/1.15 O3 JUQUEEN May 2014 I.Gutheil Folie 22

23 NAG Libraries NAG Fortran 77 Mark 23 and 24 on JUROPA in /usr/local/lib, Mark 22 on JUQUEEN: as module more than 1600 user-callable routines NAG C Mark 23: (JUROPA only) as module more than 1000 user-callable routines fl90 Release 4, (JUROPA only) in /usr/local/lib 43 new generic user-callable routines May 2014 I.Gutheil Folie 23

24 Parallell Libraries Threaded Parallelism MKL (JUROPA) is multi-threaded or at least thread-save usage as with sequential routines if OMP NUM THREADS not set, 8 threads used always use ifort name.f -o name -lmkl intel lp64 -lmkl intel thread -lmkl core -liomp5 -lpthread ESSLsmp 5.1 (JUQUEEN) Usage: mpixlf90 r name.f -L/bgsys/local/lib -lesslsmpbg FFTW 3.3 (Fastest Fourier Transform of the West) On JUQUEEN threaded and OpenMP version May 2014 I.Gutheil Folie 24

25 Parallell Libraries MPI Parallelism ScaLAPACK (Scalable Linear Algebra PACKage) ELPA (Eigenvalue SoLvers for Petaflop-Applications) Elemental, C++ framework for parallel dense linear algebra FFTW (Fastest Fourier Transform of the West) MUMPS (MUltifrontal Massively Parallel sparse direct Solver) ParMETIS (Parallel Graph Partitioning) hypre (high performance preconditioners) May 2014 I.Gutheil Folie 25

26 MPI Parallelism (II) PARPACK (Parallel ARPACK), Eigensolver planned SPRNG (Scalable Parallel Random Number Generator) SUNDIALS (SUite of Nonlinear and DIfferential/ALgebraic equation Solvers) Parallel Systems, MPI Parallelism PETSc, toolkit for partial differential equations May 2014 I.Gutheil Folie 26

27 ScaLAPACK JUROPA: part of MKL JUQUEEN: ScaLAPACK Release 2.0.2, contains already BLACS FORTRAN, also C-Interface, scalapack.h incomplete LAPACK has to be linked, too, $LAPACK DIR set together with scalapack May 2014 I.Gutheil Folie 27

28 Contents of ScaLAPACK Parallel BLAS 1-3, PBLAS Version 2 Dense linear system solvers Banded linear system solvers Solvers for Linear Least Squares Problem Singular value decomposition Eigenvalues and eigenvectors of dense symmetric/hermitian matrices May 2014 I.Gutheil Folie 28

29 Usage on JUROPA Linking a program name.f calling routines from ScaLAPACK, default version: mpif77 name.f -lmkl scalapack lp64 -lmkl blacs intelmpi lp64 -lmkl lapack -lmkl intel lp64 -lmkl intel thread -lmkl core -liomp5 -lpthread Intel compiler version and later: module unload mkl module switch intel intel/ (for example) mpif77 name.f -lmkl scalapack lp64 -lmkl blacs intelmpi lp64 -lmkl intel lp64 -lmkl intel thread -lmkl core -liomp5 -lpthread May 2014 I.Gutheil Folie 29

30 Usage on JUQUEEN Compilation and linking of a program name.f calling ScaLAPACK routines: module load scalapack/2.0.2[ g][ simd] mpixlf90 r name.f -L$SCALAPACK LIB -lscalapack -L$LAPACK LIB -llapack -L/bgsys/local/lib -lessl[smp]bg May 2014 I.Gutheil Folie 30

31 ELPA Eigenvalue SoLvers for Petaflop-Applications ELPA uses ScaLAPACK, must be linked together with scalapack FORTRAN 95, same data-distribution as ScaLAPACK http: //elpa.rzg.mpg.de/elpa-english?set_language=en JUROPA development version from November 2012, compiled with intel and version from November 2013 compiled with intel , allows hybrid parallelization JUQUEEN MPI and hybrid version from November 2013 May 2014 I.Gutheil Folie 31

32 Elemental C++ framework, two-dimensional data distribution element by element JUROPA version 0.81 compiled with intel/13.1.3, pure MPI version and hybrid version JUQUEEN 0.78-p1, MPI-only May 2014 I.Gutheil Folie 32

33 MUMPS: Multifrontal Massively Parallel sparse direct Solver Solution of linear systems with symmetric positive definite matrices, general symmetric matrices, general unsymmetric matrices Real or Complex Parallel factorization and solve phase, iterative refinement and backward error analysis F90 and MPI Version 4.8.4, 4.9.2, and on JUROPA, version on JUQUEEN May 2014 I.Gutheil Folie 33

34 ParMETIS Parallel Graph Partinioning and Fill-reducing Matrix Ordering developed in Karypis Lab at the University of Minnesota Version 3.1.1, 3.2.0, and on JUROPA, and on JUQUEEN http: //glaros.dtc.umn.edu/gkhome/metis/parmetis/overview Hypre High performance preconditioners Version 2.0.0, 2.6.0b,2.7.0b, and 2.8.0b on JUROPA, 2.8.0b and 2.9.0b, also version with bigint, on JUQUEEN, bigint cannot be used together with essl May 2014 I.Gutheil Folie 34

35 FFTW Version 2.1.5, this old version contains an MPI-parallel version of FFTW on JUROPA and JUQUEEN Version 3.2 and 3.3 on JUROPA, compiled with intel/ Version 3.3 on JUQUEEN May 2014 I.Gutheil Folie 35

36 PARPACK ARPACK Version 2.1 and PARPACK MPI-Version Must be linked with LAPACK and BLAS Reverse communication interface, user has to supply parallel matrix-vector multiplication May 2014 I.Gutheil Folie 36

37 SPRNG The Scalable Parallel Random Number Generators Library for ASCI Monte Carlo Computations Version 2.0: various random number generators in one Library Version 1.0 seperate library for each random number generator Sundials (CVODE) Package for the solution of ordinary differential equations, Version 2.3.0, 2.4.0, and on JUROPA, on JUQUEEN https: //computation.llnl.gov/casc/sundials/main.html May 2014 I.Gutheil Folie 37

38 PETSc Portable, Extensible Toolkit for Scientific Computation Numerical solution of partial differential equations version on JUROPA and on JUQUEEN basic real and complex versions advanced versions with download other packages on JUQUEEN also version with 8-Byte integer Usage at Research Centre Juelich: module avail petsc module help petsc/[whatever version you want] May 2014 I.Gutheil Folie 38

39 Software for Materials Science Simulation Available on Package JUQUEEN JUROPA Workstations ADF yes Amber yes CP2K yes yes CPMD yes yes Dalton yes Gromacs yes yes GPAW planned yes LAMMPS yes yes Molpro yes yes NAMD yes yes NWChem yes Tremolo yes TURBOMOLE yes May 2014 I.Gutheil Folie 39

40 Further informations and JSC-people Support/Software/Software_node.html Mailto I.Gutheil: Parallel mathematical Libraries J.Mextorf: NAG libraries F.Janetzko: Software for materials science B.Körfgen: Physics and Engineering software Software: May 2014 I.Gutheil Folie 40