RA MPI Compilers Debuggers Profiling. March 25, 2009

Transcription

1 RA MPI Compilers Debuggers Profiling March 25, 2009

2 Examples and Slides To download examples on RA 1. mkdir class 2. cd class 3. wget 4. tar -xzf examples.tgz 5. cd stommel Slides Note: There is summary of all scripts given at the end of the slides for easy copy/paste

3 Experimental MPI Versions

4 New MPI Compilers Version MVAPICH2 1.2 MVAPICH 1.1 OpenMPI Both Intel and Portland Group Compilers Support for Debuggers Support for Profiling

5 Need to modify your Environment Change.tcshrc or.bashrc file Log out then log back in Changes override mpi_selector settings May need to change your PBS script

6 .tcshrc settings setenv MPI_VERSION /lustre/home/apps/mpi/db/mvapich-1.1 setenv MPI_VERSION /lustre/home/apps/mpi/db/mvapich2-1.2 setenv MPI_VERSION /lustre/home/apps/mpi/db/openmpi1.3.1 setenv MPI_COMPILER intel #setenv MPI_COMPILER pg if ( $?MPI_COMPILER && $?MPI_VERSION ) then setenv MPI_BASE $MPI_VERSION/$MPI_COMPILER setenv LD_LIBRARY_PATH $MPI_BASE/lib:$LD_LIBRARY_PATH setenv LD_LIBRARY_PATH $MPI_BASE/lib/shared:$LD_LIBRARY_PATH setenv MANPATH $MPI_BASE/man:$MPI_BASE/shared/man:$MANPATH set path = ( $MPI_BASE/bin $path ) endif

7 .bashrc settings export MPI_VERSION=/lustre/home/apps/mpi/db/mvapich-1.1 export MPI_VERSION=/lustre/home/apps/mpi/db/mvapich2-1.2 export MPI_VERSION=/lustre/home/apps/mpi/db/openmpi1.3.1 export MPI_COMPILER=intel #export MPI_COMPILER=pg if [ -n $MPI_COMPILER ]; then if [ -n $MPI_VERSION ]; then export MPI_BASE=$MPI_VERSION/$MPI_COMPILER export LD_LIBRARY_PATH=$MPI_BASE/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$MPI_BASE/lib/shared:$LD_LIBRARY_PATH export MANPATH=$MPI_BASE/man:$MPI_BASE/shared/man:$MANPATH export PATH=$MPI_BASE/bin:$PATH fi fi

8 Base Script #!/bin/csh #PBS -l nodes=2:ppn=8 #PBS -l walltime=00:02:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V # cd $PBS_O_WORKDIR sort -u $PBS_NODEFILE > mynodes.$pbs_jobid ADD YOUR MPI RUN COMMAND HERE

9 MPI Run commands Version Command openmpi1.3.1 mpiexec -np 16 stc_06 mvapich2-1.2 mpiexec -np 16 /lustre/home/tkaiser/examples/stommel/stc_06 < st.in mvapich-1.1 mpirun_rsh -hostfile $PBS_NODEFILE -np 16 stc_06 < st.in mpirun -machinefile $PBS_NODEFILE -np 16 stc_06 < st.in

10 Debugging with ddt

11 Not a big fan of debuggers End up debugging the debugger Steep learning curve Can be misleading Difficult for large processor count and the problem might only show up there My favorite debuggers are printf write

12 However... I recently used ddt to find a problem for which printf did not work. It might have taken me weeks. Print statements might make the problem go away Debuggers are useful for learning a program that you have never seen ddt is working well on RA

13 Allinea DDT debugger X-Windows based ssh -X ra An initial setup is done the first time you run Works with both Portland Group and Intel Fortran Good support for Fortran modules Syntax highlighting

14 .tcshrc Environment for ddt set path = ( /lustre/home/apps/ddt2.4.1/bin $path ) setenv DMALLOCPATH /lustre/home/apps/ddt2.4.1 setenv DMALLOC setenv LD_LIBRARY_PATH $DMALLOCPATH/lib/64:$LD_LIBRARY_PATH.bashrc Requires that you use a MPI that supports debugging such as those listed above export PATH=/lustre/home/apps/ddt2.4.1/bin:$PATH export DMALLOCPATH=/lustre/home/apps/ddt2.4.1 export DMALLOC="" export LD_LIBRARY_PATH=$DMALLOCPATH/lib/64:$LD_LIBRARY_PATH

15 Debug Compile Line mpicc -g \ -L/lustre/home/apps/gdb-6.8/lib64 \ -liberty \ stc_06.c \ -o stc_06.g

16 Debug Compile Line mpicc -g -L/lustre/home/apps/gdb-6.8/lib64 -liberty \ stc_06.c \ /lustre/home/apps/ddt2.4.1/lib/64/libdmalloc.a -o \ stc_06.g Here we link to the debug memory library. This is required if you want to track memory usage in ddt. Note it must library be last on the list.

17 stdin stdout stderr stdin works for both Intel and Portland Group stdout works with the Intel compiler without modification Portland Group compiler requires a special call to be able to see stdout while the program is running, (before MPI_Init) This is NOT a bug call setvbuf3f(6,2,0) setbuf(stdout,null); for Fortan for C

18 Initial ddt setup Run first time, creates a directory ~/.ddt type ddt Choose a MPI version Choose a list of nodes (Default) Note location of this file Need to change this list to connect to running process Wait a few seconds

19 Snapz Pro X

20 Running ddt Select Run and Debug a Program Set number of processes Most likely Set threads to off Click Run Details to follow... Select the program that you will run

21 To show you... Routine required for correct stdio with Portland Group compiler Setting stdin Module support Changing values Locals / Current Line

22 Option: Let ddt submit a batch job Your run script becomes a template which ddt fills in the arguments at submit time Tell ddt the particulars Program Input # processors <= 16 ddt will watch the queue for your job to start and then connect

23 Let ddt submit a batch job Change your run line to run ddt with your program as an argument mpiexec -n 8 stf_03.g < st.in for example, becomes mpiexec -n NUM_PROCS_TAG DDTPATH_TAG/bin/ddt-debugger DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG Add (Not required but useful for attaching to already running jobs) sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes

24 A simple script (more later for specific versions of MPI) #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:10:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V # cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes #for openmpi #mpiexec -n 8 stf_03.g < st.in Note this line is commented out. This one is alive #for openmpi and ddt mpiexec -n NUM_PROCS_TAG DDTPATH_TAG/bin/ddt-debugger \ DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG

25 Under Session - Options

26 Finally select Session - New Session - Run

27 Let ddt submit the job for you

28

29 OpenMPI Debug Script #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:10:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V # cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes DDTPATH_TAG/bin/ddt-client DDT_DEBUGGER_ARGUMENTS_TAG mpiexec -np \ NUM_PROCS_TAG EXTRA_MPI_ARGUMENTS_TAG PROGRAM_TAG \ PROGRAM_ARGUMENTS_TAG

30 MVAPICH2 Debug Script #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:10:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V # cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes mpiexec -n NUM_PROCS_TAG \ DDTPATH_TAG/bin/ddt-debugger \ DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG

31 MVAPICH-1.1 Debug Script #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:15:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes mpirun_rsh -hostfile $PBS_NODEFILE -n \ NUM_PROCS_TAG DDTPATH_TAG/bin/ddt-debugger \ DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG

32 Attaching to a batch job Key here is that ddt needs to know where your job is running Add the following two lines to your script sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes ddt will look in ~/.ddt/nodes for nodes to search

33 Attaching to a batch job

34 To Attach to a Running Process Session - New Session - Attach List should pop up Nodes need to be in ~/.ddt/nodes

35

36 Attaching to a interactive job Key here is that ddt needs to know where your job is running ddt will look in ~/.ddt/nodes for nodes to search You may need to manually edit this file

37 Attaching to an interactive job

38

39 Things to show... Changing MPI version Basic setup Setting break points Seeing modules Memory usage Launching a parallel job Seeing and changing variables

40 Profiling with IPM

41 Integrated Performance Monitoring (IPM) Developed by Nick Wright of SDSC Local limited documentation Available on RA for Experimental versions of MVAPICH* Normal Compile - adding IPM library Normal MPI run Summary of MPI stats at the end of your run to stdout Can Generate a Web page with nice pictures

42 Integrated Performance Monitoring (IPM) Integrated Performance Monitoring (IPM) is a tool that allows users to obtain a concise summary of the performance and communication characteristics of their codes. IPM is invoked by the user at the time a job is run. By default, a short, text-based summary of the code's performance is provided, and a more detailed Web page summary with graphs to help visualize the output can also be obtained.

43 Environment Additions for IPM.tcshrc set path = ( $path /lustre/home/apps/pl/bin ) set path = ( $path /lustre/home/apps/ipm/bin ) setenv IPM_KEYFILE /lustre/home/apps/ipm/ipm_key.bashrc export PATH=$PATH:/lustre/home/apps/pl/bin export PATH=$PATH:/lustre/home/apps/ipm/bin export IPM_KEYFILE=/lustre/home/apps/ipm/ipm_key

44 Compiling for IPM mpif90 -g stf_03.f90 -L$MPI_BASE/ipm/lib -lipm -o stf_03.ipm $MPI_BASE = /lustre/home/apps/mpi/db/version VERSION mvapich-1.1/pg mvapich-1.1/intel mvapich2-1.2/pg mvapich2-1.2/intel openmpi/* Works? yes Stay Tuned yes yes No - know problem

45 ##IPMv0.923#################################################################### # # command : unknown (completed) # host : compute-9-9/x86_64_linux mpi_tasks : 8 on 1 nodes # start : 03/24/09/14:08:52 wallclock : sec # stop : 03/24/09/14:09:24 %comm : 1.24 # gbytes : e+00 total gflop/sec : e+00 total # ############################################################################## # region : * [ntasks] = 8 # # [total] <avg> min max # entries # wallclock # user # system # mpi # %comm # gflop/sec # gbytes # # # [time] [calls] <%mpi> <%wall> # MPI_Recv # MPI_Reduce # MPI_Send # MPI_Bcast # MPI_Comm_size # MPI_Allreduce # MPI_Allgather # MPI_Comm_rank e ###############################################################################

46 3/24/09 2:15 PM Generate a web page: ipm_parse -html tkaiser IPM profile for unknown IPM profile for unknown 3/24/09 2:15 PM IPM profile for unknown 3/24/09 2:15 PM unknown Load Balance Communication Balance Message Buffer Sizes Communication Topology Switch Traffic Memmory Usage Executable Info Host List Environment Developer Info command: unknown codename: unknown state: running username: tkaiser group: tkaiser host: Computation compute-9-9 (x86_64_linux) mpi_tasks: 8 on 1 hosts start: 03/24/09/14:08:52 wallclock: e+01 sec stop: 03/24/09/14:09:24 %comm: total memory: 0 gbytes total gflop/sec: switch(send): 0 gbytes switch(recv): 0 gbytes Communication Event Count Pop NULL 0 * % of MPI Time by MPI rank, by MPI time Load balance by task: memory, flops, timings by MPI rank, time detail by MPI time, time detail by rank, call list Message Buffer Size Distributions: time IPM profile for unknown 3/24/09 2:15 PM HPM Counter Statistics Event Ntasks Avg Min(rank) Max(rank) NULL * (0) 0 (0) Communication Event Statistics (100.00% detail, e-06 error) Buffer Size Ncalls IPM Total profile Time for unknown Min Time Max Time %MPI %Wall MPI_Recv e e MPI_Recv e e MPI_Reduce e e MPI_Send e e MPI_Send e e Load balance by task: HPM counters 3/24/09 2:15 PM by MPI rank, by MPI time Communication balance by task (sorted by MPI time) cumulative values, values Message Buffer Size Distributions: Ncalls file:///users/tkaiser/desktop/unknown_8_tkaiser _ipm_unknown/index.html Page 1 of 5 file:///users/tkaiser/desktop/unknown_8_tkaiser _ipm_unknown/index.html Page 2 of 5 file:///users/tkaiser/desktop/unknown_8_tkaiser _ipm_unknown/index.html Page 3 of 5 cumulative values, values Communication Topology : point to point data flow data sent, data recv, time spent, map_data file map_adjacency file Switch Traffic (volume by node) Memory usage by node file:///users/tkaiser/desktop/unknown_8_tkaiser _ipm_unknown/index.html Page 5 of 5

47 Can profile sections Report will have a new page with the given label!turn on profiling call mpi_pcontrol( 1,"proc_a"//char(0))...!turn off profilingcall mpi_pcontrol( -1,"proc_a"//char(0)) /* turn on profiling*/ MPI_Pcontrol( 1,"proc_a");... /* turn off profiling*/ MPI_Pcontrol(-1,"proc_a");

48 What s Missing What are we doing about it? Timeline style program tracing Time in MPI routines Communication patterns Time in other routines Memory Tracking Performance numbers Flops Cache misses...

49 Tracing Evaluated a commercial package and rejected it Will be installing Tau Large package which does preprocessing of source Works with many analysis packages Includes memory tracking if malloc/allocate can be seen

50 Performance Information Some Examples: Software/Tools/PAPI/ x50.html How do we get it? PAIP

51 PAPI - Performance API Specifies a standard application programming interface (API) for accessing hardware performance counters available on most modern microprocessors Used by both Tau and IPM Can show the effects of different optimizations Problem: requires Kernel Patch

52 Tau and PAPI part of POINT Productivity from Open, INtegrated Tools (POINT) project is funded as part of the NSF's Software Development for Cyberinfrastructure (SDCI) program Goal: integrate, harden, and deploy an open, portable, robust performance tools environment

53 Summary The DDT debugger is available for parallel applications DDT can also track memory usage IPM is currently available for simple profiling We will be installing additional performance analysis tools Summary of scripts follows...

54 .tcshrc additions summary ### mpi settings ## setenv MPI_VERSION /lustre/home/apps/mpi/db/mvapich-1.1 setenv MPI_VERSION /lustre/home/apps/mpi/db/mvapich2-1.2 setenv MPI_VERSION /lustre/home/apps/mpi/db/openmpi1.3.1 setenv MPI_COMPILER intel #setenv MPI_COMPILER pg if ( $?MPI_COMPILER && $?MPI_VERSION ) then setenv MPI_BASE $MPI_VERSION/$MPI_COMPILER setenv LD_LIBRARY_PATH $MPI_BASE/lib:$LD_LIBRARY_PATH setenv LD_LIBRARY_PATH $MPI_BASE/lib/shared:$LD_LIBRARY_PATH setenv MANPATH $MPI_BASE/man:$MPI_BASE/shared/man:$MANPATH set path = ( $MPI_BASE/bin $path ) endif ### ddt settings ### set path = ( /lustre/home/apps/ddt2.4.1/bin $path ) setenv DMALLOCPATH /lustre/home/apps/ddt2.4.1 setenv DMALLOC setenv LD_LIBRARY_PATH $DMALLOCPATH/lib/64:$LD_LIBRARY_PATH ### ipm settings ### set path = ( $path /lustre/home/apps/pl/bin ) set path = ( $path /lustre/home/apps/ipm/bin ) setenv IPM_KEYFILE /lustre/home/apps/ipm/ipm_key

55 .bashrc additions summary ### mpi settings ### export MPI_VERSION=/lustre/home/apps/mpi/db/mvapich-1.1 export MPI_VERSION=/lustre/home/apps/mpi/db/mvapich2-1.2 export MPI_VERSION=/lustre/home/apps/mpi/db/openmpi1.3.1 export MPI_COMPILER=intel #export MPI_COMPILER=pg if [ -n $MPI_COMPILER ]; then if [ -n $MPI_VERSION ]; then export MPI_BASE=$MPI_VERSION/$MPI_COMPILER export LD_LIBRARY_PATH=$MPI_BASE/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$MPI_BASE/lib/shared:$LD_LIBRARY_PATH export MANPATH=$MPI_BASE/man:$MPI_BASE/shared/man:$MANPATH export PATH=$MPI_BASE/bin:$PATH fi fi ### ddt settings ### export PATH=/lustre/home/apps/ddt2.4.1/bin:$PATH export DMALLOCPATH=/lustre/home/apps/ddt2.4.1 export DMALLOC="" export LD_LIBRARY_PATH=$DMALLOCPATH/lib/64:$LD_LIBRARY_PATH ### ipm settings ### export PATH=$PATH:/lustre/home/apps/pl/bin export PATH=$PATH:/lustre/home/apps/ipm/bin export IPM_KEYFILE=/lustre/home/apps/ipm/ipm_key

56 Compiling for IPM mpif90 -g stf_03.f90 -L$MPI_BASE/ipm/lib -lipm -o stf_03.ipm $MPI_BASE = /lustre/home/apps/mpi/db/version VERSION mvapich-1.1/pg mvapich-1.1/intel mvapich2-1.2/pg mvapich2-1.2/intel openmpi/* Works? yes Stay Tuned yes yes No - know problem

57 Debug Compile Line mpicc -g \ -L/lustre/home/apps/gdb-6.8/lib64 \ -liberty \ stc_06.c \ -o stc_06.g

58 Debug Compile Line mpicc -g -L/lustre/home/apps/gdb-6.8/lib64 -liberty \ stc_06.c \ /lustre/home/apps/ddt2.4.1/lib/64/libdmalloc.a -o \ stc_06.g Here we link to the debug memory library. This is required if you want to track memory usage in ddt. Note it must library be last on the list.

59 OpenMPI Debug Script #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:10:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V # cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes DDTPATH_TAG/bin/ddt-client DDT_DEBUGGER_ARGUMENTS_TAG mpiexec -np \ NUM_PROCS_TAG EXTRA_MPI_ARGUMENTS_TAG PROGRAM_TAG \ PROGRAM_ARGUMENTS_TAG

60 MVAPICH2 Debug Script #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:10:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V # cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes mpiexec -n NUM_PROCS_TAG \ DDTPATH_TAG/bin/ddt-debugger \ DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG

61 MVAPICH-1.1 Debug Script #!/bin/csh #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:15:00 #PBS -N testio #PBS -o stdout.$pbs_jobid #PBS -e stderr.$pbs_jobid #PBS -r n #PBS -V cd $PBS_O_WORKDIR #save a nicely sorted list of nodes sort -u $PBS_NODEFILE > mynodes.$pbs_jobid cp mynodes.$pbs_jobid ~/.ddt/nodes mpirun_rsh -hostfile $PBS_NODEFILE -n \ NUM_PROCS_TAG DDTPATH_TAG/bin/ddt-debugger \ DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG