WP7: Adaptive Grid Components

Transcription

1 WP7: Adaptive Grid Components a status update Thilo Kielmann kielmann@cs.vu.nl Vrije Universiteit, Amsterdam Thilo Kielmann GridLab Conference, Eger 1/17

2 Goals of WP7 Provide adaptive application components to GAT modules Focus core adaptation mechanisms inside one re-usable implementation: retrieving monitoring information correlation of monitoring data and application performance forecasting Define API and event model specific per use case Thilo Kielmann GridLab Conference, Eger 2/17

3 Collaboration with WPs WP1 GAT WP2 Cactus GAT WP3 Triana GAT WP8 Data handling and Visualization WP9 Resource Management WP7 Adaptive Components WP10 Information Services WP11 Monitoring WP7-components interface the GAT to resource information Thilo Kielmann GridLab Conference, Eger 3/17

4 Use Cases Throughput optimization of Triana flow graphs (1) problem: placement of Triana units to compute nodes resources: CPU/JVM speed 2. Throughput optimization of Triana flow graphs (2) problem: unit placement, limited communication speed resources: CPU/JVM speed, network bandwidth 3. Parameter adaptation for parallel I/O problem: select file system and I/O stride resources: I/O bandwidth from CPUs to local disks Thilo Kielmann GridLab Conference, Eger 4/17

5 Use Cases N-to-M data transfer time estimation problem: migrating user data from N to M nodes resources: network bandwidth 5. Remote data visualization problem: trading image quality against frame rate resources: network bandwidth 6. Communication adaptation of distr. simulations (1) problem: optimize ghost zones for simulation data resources: network latency, bandwidth, CPU speed Thilo Kielmann GridLab Conference, Eger 5/17

6 Use Cases Communication adaptation of distr. simulations (2) problem: optimize ghost zones with compression resources: network latency, bandwidth, CPU speed 8. Distributing computation across heterogeneous CPUs problem: data distribution among CPU nodes resources: CPU speed 9. Compute time estimation problem: prediction of job completion times resources: CPU speed, queue status (used for the scenario from the Zakopane workshop) Thilo Kielmann GridLab Conference, Eger 6/17

7 Adaptation Component Idea: Adaptation Component monitor control Application Thilo Kielmann GridLab Conference, Eger 7/17

8 Adaptation Component Refined: User/ Portal requirements status info Adaptation Component monitor monitor control Application Network CPU/Disk Thilo Kielmann GridLab Conference, Eger 8/17

9 Core Mechanisms data event examples adaptation API / event model observed app. level state predicted app. level state time per iteration memory footprint correlation app. level performance model raw data monitored prediction raw data predicted TCP bandwidth free memory Thilo Kielmann GridLab Conference, Eger 9/17

10 Important Resources Identified via our use cases: 1. compute speed (application specific, also queue waiting times) 2. network performance (bandwidth, latency, topology, Path-MTU) 3. I/O bandwidth (mostly local) We need to monitor them! Thilo Kielmann GridLab Conference, Eger 10/17

11 WP11 Monitoring System 1. Generic framework for monitoring of various characteristics 2. Can support us with CPU utilization queue waiting/completion times application progress 3. Current implementation suited for passive probing 4. However, network monitoring is mostly active! Thilo Kielmann GridLab Conference, Eger 11/17

12 Network Characteristics From GGF s NM WG: Hoplist Characteristic Bandwidth Queue Capacity Utilized Discipline Available Achievable Capacity Length Delay Forwarding Policy Forwarding Round trip One way Jitter Forwarding Table Forwarding Weight Loss Loss Pattern Availability Round trip One way MTBF Avail. Pattern Packet Reordering Reordering Pattern Closeness Thilo Kielmann GridLab Conference, Eger 12/17

13 Characteristics we use Characteristic Hoplist TopoMon Bandwidth Capacity Available Pathrate Utilized Achievable Delay NWS Round trip Thilo Kielmann GridLab Conference, Eger 13/17

14 The Adaptive Service GAT adaptive components, part of GAT modules: adaptation requests Web Service (Grid Service) "Adaptive Service" ACS adaptive components service adaptation requests adaptation requests historic data storage LAC System 1 historic data storage LAC System 2 MM wp11 NWS TopoMon MM wp11 NWS TopoMon Thilo Kielmann GridLab Conference, Eger 14/17

15 Network Visualization visualization cache caching client adaptation requests Web Service (Grid Service) "Adaptive Service" ACS adaptive components service adaptation requests adaptation requests historic data storage LAC System 1 historic data storage LAC System 2 MM wp11 NWS TopoMon MM wp11 NWS TopoMon Thilo Kielmann GridLab Conference, Eger 15/17

16 Plans for Eger Workshop Deploy network monitoring on more sites with WP8: work on data-related use cases (n-to-m transfer, remote visualization) with WP11: further integration with all: taste the local wine! Thilo Kielmann GridLab Conference, Eger 16/17

17 Thanks to: Rob van Nieuwpoort, Jason Maassen: Adaptive Service, NWS/Monitoring Integration Mathijs den Burger: TopoMon, Caching Client Zuzaná Bizoňová: Network Visualization Kees Verstoep: Globus and Firewall Gymnastics European Commission: Money :-) Thilo Kielmann GridLab Conference, Eger 17/17