A Performance Monitor based on Virtual Global Time for Clusters of PCs

Transcription

1 A Performance Monitor based on Virtual Global Time for Clusters of PCs Michela Taufer Scripps Institute & UCSD Dept. of CS San Diego, USA Thomas Stricker Cluster 2003, 12/2/2003 Hong Kong, SAR, China Lab. for Computer Systems Swiss Institute of Technology ETH Zürich, Switzerland Slides of this talk: Eidgenössische Technische Hochschule Zürich 1

2 Work motivated by a study of distributed DB on cluster of commodity PCs OLAP Application TPC-D Middleware TP-Lite ORACLE Distributed system LINUX PC LINUX PC LINUX PC LINUX PC LINUX PC LINUX PC 3 2

3 TPC-D benchmark for OLAP (data-mining) TPC-D data model Customer Nation Region Database size: 10 GByte Order Supplier Part PartSupp Fully replicated data LineItem Disjointedly distributed data 3

4 Results in Performance and Scalability Scalability Optimal speed-up Slow down We look into the resource usage of the TPC-D benchmark to explain the scalability picture 4 4

5 Motivation Outline Keeping a performance monitor in distributed systems with middleware packages Complementing middleware with inverted middleware Issues addressed and problems solved in the inverted middleware performance monitoring approach application performance model based on execution time and machine resource usage keeping track of computation phases in the application using a notion of global virtual time dealing with monitoring intrusion by dropping samples (UDP/IP) Experimental results - see our tool at work Conclusions 5

6 Using Middleware in Clusters Benefits of middleware packages (DBMS in this case): provide higher level of abstraction for programmer address the problem of distribution to nodes hide system dependencies Drawbacks of using middleware packages: make instrumentation for performance analysis related to distribution of computation in cluster difficult obstruct the detection of performance bottlenecks and architectural problems in the distributed system 6

7 Our Approach: Complementing middleware (MW) with inverted middleware (MW -1 ) middleware with inverted middleware 7

8 Inverted Middleware as a Performance Monitoring Tool The relation between middleware and inverted middleware as a performance monitoring tool is similar to the relation between compiler and debugger. Inverted middleware comprises: software instrumentation at the OS level to gather data at the nodes of a cluster. infrastructure for the collection of performance data. analytical model for reconstructing a global picture of application performance. 8

9 Architecture of Inverted Middleware as Cluster Monitoring Tool Our monitoring tool has a master-worker setting worker worker resources sample sample resources daemon daemon data collector data files control controller monitoring master 9

10 Internal Structure of Inverted Middleware Application MW MW -1 Application-specific layer: uses the global performance data for optimization and prediction Distribution-specific layer: collects performance data from the several nodes and patches it into a coherent view Distributed System System-specific layer: samples system specific performance data on resource usage and bottlenecks 10

11 Performance Data Collected Packet identification: Worker_id, packet_id (counter) CPU performance data: Total number of user instructions over the interval of time t Total number of system instructions over the interval of time t Disks performance data: Total number of bytes read and written per disks (we have three disks) over the time interval t Total number of bytes accessed sequentially/bytes accessed not sequentially per disks over the time interval t Network performance data: Number of packets/bytes sent and received on the several network connections over the interval of time t 11

12 System-Specific Layer (sampling local performance data) performance sample CPU data network data disk data CPU monitoring library kernel module hardware performance counters /proc file system Linux kernel disk registers kernel hooks 12

13 System-Specific Layer System S ecific Layer Sampling mechanism Dynamic sample of resource information (e.g. floating point operations, amount of traffic over the network or to the disks) at regular intervals Sampling performance data Outside the kernel as daemon processes Performance hooks into extended LINUX /proc file system and hardware performance counters provided by the processors No re-engineering of application code or middleware 13

14 Distribution-Specific Layer worker resources sample daemon application subjob UDP/IP TCP/IP collector controller data files monitoring master application master Monitoring and application masters can reside on different PCs 14

15 Application-Specific Layer data control collector controller data files data monitoring master data eater Application-specific model global system overview 15

16 Application-Specific Layer Performance model of the application: translates the elementary knowledge about the resource usage into high level answers to performance questions and bottlenecks suitable for suggesting optimizations to the user is a simple set of formulas which allows the calculation of the individual execution times due to the usage of each machine resource needs some calibration and validation 12 16

17 Our Simple Performance Model Different approach to system performance: Application user: Total execution time System engineer: Usage of system resources We claim: Simple relationship between the machine resources required by the application and its execution time Total execution time can be decomposed into parts Each part is largely determined by the usage of one single critical machine resource Execution time is linked to resource usage! 17

18 Metrics in Performance Model Everything is translated into execution time: CPU: Number of FlOp * Processing Rate (FlOp/s) Disk: Number of Bytes read/written * Data Rate Disk: Number of Random Access * Average Access Time Network: Number of Bytes transferred * Bandwidth Network: Number of Messages sent/rcvd * Latency/Overhead All metrics are transmitted from node to master: with timestamps to allow reconstruction of time variant load. as total count to allow reconstruction of dropped samples. per node samples to allow reconstruction of global count. 18

19 Virtual Time - Phases of an Application Coarse granularity samples for long time simulations Daemon samples resource data at regular intervals User decides the time interval t for each single node Each sample collects total resource data within the time interval t Size of the sample packet is small: less than 126KByte samples time interval t time 19

20 Virtual Time - Phases of an Application speed (MBit/s) coordinator /3 phase 1 virtual barrier ph time (sec) speed (MBit/s) virtual virtual barrier barrier node 1/3 node 2/3 node 3/ time (sec) speed (MBit/s) time (sec) speed (MBit/s) virtual barrier time (sec) send to nodes receive from coordinator receive from nodes send to coordinator Computation Phase: CPU limited Communication Phase: network limited 20

21 Managing Intrusion of Monitoring Traffic Protocol for the monitoring traffic: UDP/IP Loss of monitoring data packets Treatment of lost information as sample errors Loose notion of time [Fox] Strong synchronization at the start of monitoring session Build-in cycle counters as virtual barriers for the synchronization of the samples Master-slave paradigm of the performance data collection: Free allocation of monitoring master 21

22 Experiments with the Tool at Work (Explaining a Scalability Problem of TPC-D query 3 as Resource Bottleneck) Relative Execution Speed Speed up on Cluster of 3 and 6 nodes ideal ideal Number of Processors Fast Gigabit Ethernet Ethernet 22

23 Communication-Limited Queries: Resource Usage 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% number of nodes Fast Ethernet Gigabit Ethernet cpu time disk time (non-seq access) communication time disk time (seq access) Fast Ethernet vs. Gigabit Ethernet 23

24 Does it always work... (what about queries with disastrous performance?) allocated to machine-resources non-allocated to machine-resources CPU time communication time disk time (no seq access) disk time (seq access) Most time is spent accessing data to the disks in a non-sequential way 24

25 Target of Performance Studies: Study of Different Platforms Platform Factors Focus on machine resources: CPU clock rate factor: Slow rate (400 MHz) High rate (1 GHz) Disk rate factor: Slow disks (7.3 ms/22 MB/s) Fast disks (6.8 ms/30 MB/s) Network technology factor: Fast Ethernet (100 Mb/s) Gigabit Ethernet (1000 Mb/s) Slow disks Fast disks 1GHz 400MHz CPU clock rate Fast Ethernet Gigabit Ethernet Networking technology Cluster size: 1, 3, 6 nodes Disk rate 15 25

26 Conclusions Performance monitoring with middleware in distributed systems such as clusters and desktop grids remains difficult. Our attempt to build such a tool more systematically and more scientifically lead to: A new view of the performance debugging tool as inverted middleware leads to a modular and layered design of tools. A simple, but effective performance model directly connects execution time to the performance critical machine resources (CPU, disks, network) during different phases of an application. A technique to keep intrusions low uses unreliable communication (UDP/IP) to collect samples and a notion of global virtual time to reconstruct data dropped in network congestion. Much more work is needed in performance monitoring for clusters. We are far from tools that are as clear in functionality and as straight forward to use as e.g. gcc, gdb or gprof. 26