Debugging with (and without) TotalView

Transcription

1 Debugging with (and without) TotalView Dieter an Mey Center for Computing and Communication Aachen University of Technology 1 TotalView, VI-HPS Tuning Workshop 2008

2 Help TotalView is a commercial debugger of Etnus Inc.: TotalView Users Guide TotalView Online Help Appendix in RWTH Primer Online Tutorial LLNL TotalView, VI-HPS Tuning Workshop 2008

3 Start of TotalView Current Version is 8.x Compilewith-g and without optimization Start executable with Serial: totalview a.out Serial: totalview a.out a args Sun MPI (HPC CT 6): totalview mprun -a -np 8... a.out Sun MPI (HPC CT 6): totalview mprun -a -np 8... a.out args Intel MPI: mpiexec -tv -np 8... a.out args Investigate a core file: totalview a.out core Attach to a running program: totalview 3 TotalView, VI-HPS Tuning Workshop 2008

4 TotalView Windows - Overview Root Window Variable Window Stack Trace Stack Frame Process Window Expression List Terminal Session 4 TotalView, VI-HPS Tuning Workshop 2008 Action Points

5 Root Window 5 TotalView, VI-HPS Tuning Workshop 2008

6 Status + ID Process Window Stack Trace Stack + Register Threads Action Points 6 TotalView, VI-HPS Tuning Workshop 2008

7 Variable Window (1) RMB: Dive in New Window Type casting 7 TotalView, VI-HPS Tuning Workshop 2008

8 Variable Window (2) Slicing Filtering 8 TotalView, VI-HPS Tuning Workshop 2008

9 Surface View Window Tools - Visualize 9 TotalView, VI-HPS Tuning Workshop 2008

10 Surface View Window 10 TotalView, VI-HPS Tuning Workshop 2008

11 Watchpoint if ( k%50==0 ) { $visualize (u); $stop; } 11 TotalView, VI-HPS Tuning Workshop 2008

12 OpenMP - Debugging 12 TotalView, VI-HPS Tuning Workshop 2008

13 Debugging OpenMP Programs? If you want to debug your OpenMP code try to avoid using TotalView Well try to avoid to use any debugger with OpenMP Debug your serial program first. And then look out for data races which may not show up during the debugging process If you still want to debug your OpenMP code you may want to use TotalView 13 TotalView, VI-HPS Tuning Workshop 2008

14 OpenMP Debugging Recepy 1 of 2 Prepare the serial code Carefully select a reasonable test case! Is the serial program delivering the right results? ( with optimization turned on? ) How about compiler warnings? Fortran: Put all local variables on the stack first. Now try the OpenMP version Need to encrease the stacksize limits? export STACKSIZE=... # not yet standardized; ulimit s... Respect compiler messages (your compiler may have switches to turn on excessive checking) Fortran: USE omp_lib Try the OpenMP dummy library, which your compiler may provide. (Sun Studio: link with lompstubs ) 14 TotalView, VI-HPS Tuning Workshop 2008

15 OpenMP Debugging Recepy 2 of 2 Is the OpenMP program running well with a single thread? Is the OpenMP program running correctly sometimes with more than one thread? Race Conditions? Thread Safety? Use of static or global variables within a parallel region? (f90: SAVE, DATA, initializations,..., C: static, extern ) Use a data race detection tool (Intel Thread Checker, Sun Studio 12) Turn on and off single parallel regions! serialise parts of long parallel regions (single directive) introduce additional barriers for testing Different rounding errors matter? Turn off certain compiler optimizations Don t parallelize reductions 15 TotalView, VI-HPS Tuning Workshop 2008

16 Data Races The typical OpenMP programming errors: Data Races One thread modifies a memory location, which another thread reads or writes in the same region (between 2 synchronisation points). Take care: The sequence of the execution of parallel loop iterations is non deterministic and may change from run to run. Necessary condition for parallelizing a loop: The serial code should give the same answers, when running the loop backwards. Data race detection tools trace memory references and detect possible data races which may never occur while you step through your code with a debugger In many cases private clauses, barriers, or critical regions are missing. 16 TotalView, VI-HPS Tuning Workshop 2008

17 Using TotalView with OpenMP See TotalView User s Guide: Each parallel region is outlined into a separate routine Each parallel loop is outlined into a separate routine The names of these outlined routines base on the original name of the calling routine and the line number of the parallel directive Shared variables are declared in the calling routine and passed to the outlined routine. Private variables are declared in the outlined routine. The slave threads are generated on entry of the parallel region You must not step into a parallel region, but run into a previously defined breakpoint. 17 TotalView, VI-HPS Tuning Workshop 2008

18 Example OpenMP-Program x = 43.0 h = 1.0d0 / n sum = 0.0d0!$omp parallel do private(i,x) reduction(+:sum) shared(n,h) do i = 1,n x = h * ( i ) sum = sum + 4.0d0 / ( 1.0d0 + x * x ) end do!$omp end parallel do pi = h * sum x =? In a parallel region (loop...), when watching a shared variable, look at the variable in the original routine. when watching a private variable, look at that variable in the outlined routine. In OpenMP V2.5 the private version of the master thread may share the memory location of the original variable. In OpenMP V3.0 this may no longer be allowed. 18 TotalView, VI-HPS Tuning Workshop 2008

19 MPI - Debugging 19 TotalView, VI-HPS Tuning Workshop 2008

20 Start Process Window Root Window 20 TotalView, VI-HPS Tuning Workshop 2008

21 Start Process Window Root Window 21 TotalView, VI-HPS Tuning Workshop 2008

22 Start Process Window Root Window 22 TotalView, VI-HPS Tuning Workshop 2008

23 Root Window Open another process window by right clicking and selecting Dive in New Window Process ID State: B Breakpoint (stopped) E Error (stopped) H Hold I Idle K in Kernel M Mixed Name R Running S Sleeping T Stopped W Watchpoint (stopped) Z Zombie 23 TotalView, VI-HPS Tuning Workshop 2008

24 Process Window Switch between processes by clicking on P- or P+ Switch between threads by clicking on T- or T+ 24 TotalView, VI-HPS Tuning Workshop 2008

25 Process Window Evaluation Point Breakpoint Barrier Conditional Watchpoint Unconditional Watchpoint 25 TotalView, VI-HPS Tuning Workshop 2008

26 Laminate Variables Variable my_rank Contains the id of each process 26 TotalView, VI-HPS Tuning Workshop 2008

27 Laminate Variables 27 TotalView, VI-HPS Tuning Workshop 2008

28 Laminate Variables 28 TotalView, VI-HPS Tuning Workshop 2008

29 Summary Debugging of Parallel codes: Parallelize carefully! Parallelization adds one dimension to the error space Let the compiler statically analyse your code + watch the messages Check the interfaces In the case of MPI There are deadlock detection tools out there: MARMOT In the case of OpenMP: Most likely, using a debugger on OpenMP codes is not necessary: Use Sun Thread Analyzer or Intel Thread Checker to detect races If you really want to debug: TotalView might help you. Never put an OpenMP Code into production without checking for data races beforehand! 29 TotalView, VI-HPS Tuning Workshop 2008