Report on Project: Advanced System Monitoring for the Parallel Tools Platform (PTP)

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Report on Project: Advanced System Monitoring for the Parallel Tools Platform (PTP) September, 2014 Wolfgang Frings and Carsten Karbach

2 Project progress Server caching Customized display layouts Optimized job localization Usage history and scheduling prediction September, 2014 Wolfgang Frings and Carsten Karbach 2 30

3 Content 1 LLview/PTP overview 2 Server caching 3 Display layouts 4 Optimized job localization 5 Embed LLview diagrams September, 2014 Wolfgang Frings and Carsten Karbach 3 30

4 Mitglied der Helmholtz-Gemeinschaft Part I: LLview/PTP overview September, 2014 Wolfgang Frings and Carsten Karbach

5 1. LLview/PTP overview LLview Visualizes supercomputer status on a single screen Source: Screenshot LLview for JUQUEEN September, 2014 Wolfgang Frings and Carsten Karbach 5 30

6 1. LLview/PTP overview PTP Parallel Tools Platform What is PTP? IDE for parallel application development Based on Eclipse Open-source project Developers: IBM, U.Oregon, UTK, Heidelberg University, NCSA, UIUC, JSC,... September, 2014 Wolfgang Frings and Carsten Karbach 6 30

7 1. LLview/PTP overview Monitoring Architecture Supercomputer Client llview-client update request LML data gatherer LML_da LML response PTP September, 2014 Wolfgang Frings and Carsten Karbach 7 30

8 Mitglied der Helmholtz-Gemeinschaft Part II: Server caching September, 2014 Wolfgang Frings and Carsten Karbach

9 2. Server caching Architecture multiple users on the same target system cache LML file in public directory (e.g. /tmp), use LML cache as data source default: each client triggers independent status data update caching: daemon retrieves status data, clients use cached data Cache workflow September, 2014 Wolfgang Frings and Carsten Karbach 9 30

10 2. Server caching Implementation LML caching is included in latest Eclipse release Luna (July 14) from Implementation is documented on bug Install daemon, once for each system 1 Create monitoring connection and start it 2 SSH to target system, switch to.eclipsesettings/util/install 3 Call install script: perl installcrontab.pl 4 Run util/install/lml da crontab.sh every minute September, 2014 Wolfgang Frings and Carsten Karbach 10 30

11 2. Server caching Install step 1 Start monitoring connection September, 2014 Wolfgang Frings and Carsten Karbach 11 30

12 2. Server caching Install steps 2, 3, 4 Install daemon and run it September, 2014 Wolfgang Frings and Carsten Karbach 12 30

13 2. Server caching Client usage Default update mode: active caching If no daemon installed no caching Users can deactivate caching via Force Data Update September, 2014 Wolfgang Frings and Carsten Karbach 13 30

14 2. Server caching Advantages of caching Faster response time System Cache [s] No Cache [s] JUDGE JUROPA JUQUEEN Recording of history data is possible Enhancement with data not accessible to normal user Decreased load for the system September, 2014 Wolfgang Frings and Carsten Karbach 14 30

15 Mitglied der Helmholtz-Gemeinschaft Part III: Display layouts September, 2014 Wolfgang Frings and Carsten Karbach

16 3. Display layouts Customized LML layouts understand system architecture and hierarchy map topology into LML layout advantages: level of detail, automatic job filtering, display node names, improved performance workplan: tutorial on LML layouts, contact system administrators of partner XSEDE/PRACE sites, support writing of LML layouts, ask for feedback September, 2014 Wolfgang Frings and Carsten Karbach 16 30

17 3. Display layouts Example JUDGE Default layout September, 2014 Wolfgang Frings and Carsten Karbach 17 30

18 3. Display layouts Example JUDGE Customized layout alternating background colors improved grid for square display highlighted GPUs September, 2014 Wolfgang Frings and Carsten Karbach 18 30

19 3. Display layouts Example JUROPA Default layout September, 2014 Wolfgang Frings and Carsten Karbach 19 30

20 3. Display layouts Example JUROPA Customized layout four level hierarchy: partition, rack, node, core level of detail (only with layout) better overview September, 2014 Wolfgang Frings and Carsten Karbach 20 30

21 3. Display layouts Example JUQUEEN Default layout September, 2014 Wolfgang Frings and Carsten Karbach 21 30

22 3. Display layouts Example JUQUEEN Customized layout colored rows do not display rack names September, 2014 Wolfgang Frings and Carsten Karbach 22 30

23 3. Display layouts Scalability 1 Scalable server scripts LML da Select only required data 2 Scalable data format LML Data compression Redundancy avoidance Uses system hierarchy 3 Scalable visualization PTP Filter status data Levels of detail Zoom function September, 2014 Wolfgang Frings and Carsten Karbach 23 30

24 Mitglied der Helmholtz-Gemeinschaft Part IV: Optimize job localization September, 2014 Wolfgang Frings and Carsten Karbach

25 4. Optimize job localization Job selection raised in bug allow selection of multiple jobs keep selected job selected until it is de-selected mark entire connected area of each job Source: September, 2014 Wolfgang Frings and Carsten Karbach 25 30

26 Mitglied der Helmholtz-Gemeinschaft Part V: Embed LLview diagrams September, 2014 Wolfgang Frings and Carsten Karbach

27 5. Embed LLview diagrams LLview diagrams histograms prediction load history Challenges Re-implementation in Java? Double update for LLview and PTP September, 2014 Wolfgang Frings and Carsten Karbach 27 30

28 5. Embed LLview diagrams First implementation step September, 2014 Wolfgang Frings and Carsten Karbach 28 30

29 5. Embed LLview diagrams Evaluation Advantages expecting little implementation effort single source of diagram implementation use of SVG graphics within PTP and web client SVG graphics can contain tags for interactive data analysis Problems and risks adding LLview-client to PTP source, license analysis of SVG performance and support September, 2014 Wolfgang Frings and Carsten Karbach 29 30

30 Contact PTP LLview September, 2014 Wolfgang Frings and Carsten Karbach 30 30