Arun Kejariwal. CS229 Project Report

Transcription

1 Arun Kejariwal CS229 Project Report

2 Abstract Elasticity of cloud assists in achieving availability of web applicatis by scaling up a cluster as incoming traffic increases and keep operatial costs under check by scaling down a cluster when incoming traffic decreases. The elasticity of cloud makes performance modeling and analysis n- trivial. In the ctext of Java- based web applicatis, a key aspect is the performance of the garbage collector (GC). Existing tools for analyzing the performance of a GC are tailored for a single Java process, hence not suitable for comparative garbage collector performance analysis across all the s in a cluster in the cloud. The variati in GC performance stems from a multitude of sources, for example (but not limited to) multi- tenancy, n- uniform arrival traffic and other systemic factors. The objective of the project is to employ unsupervised learning techniques to detect bad s (where, for example, Full GC time a bad would be much higher than the healthy s). Introducti Netflix migrated its infrastructure from a traditial data center to AWS EC2 to leverage, amgst other benefits, elasticity and support for multiple availability zes. Migrati to the cloud eliminated the cycles spent hardware procurement, datacenter maintenance and resulted in higher development agility. Having said that, the elasticity of AWS a cluster can be scaled up or down makes performance characterizati and tuning of, in the current ctext, Java- based applicatis in the cloud n- trivial. Tuning the Java HotSpot virtual machine garbage collecti (GC) is key to performance optimizati of web applicatis. For instance, Zhao et al. (OOPSLA 09) demstrated that GC has material impact the performance and scalability of Java applicatis multi- core platforms. At Netflix, and in general, the following factors, in part, mandate the need for a tool for advanced analysis and tuning GC performance in the cloud. Performance evaluati of a new JVM: Recently, Java 7 was announced. Several new optimizatis such as Compressed Oops, Escape Analysis are supported in Java 7. The new optimizatis can potentially impact GC performance. For example, escape analysis can result in eliminati of certain object allocatis. Figure 1 shows the time series of, in JCsole, of e of the Netflix applicatis. The time series does not throw light the distributi of that is needed to characterize the garbage collector performance. Figure 1 Applicati-driven selecti of garbage collector type: Compare the performance of the G1 GC, first introduced in Java and the ccurrent-mark-and-sweep CMS (default in Java 7) GC. The focus of this project is to detect outlier with respect to GC performance s in a given cluster.

3 Brief Background In this secti, we present an overview of the JVM layout and the associated terminology. Figure 2 depicts the layout of the HotSpot VM. From the Figure we note that Java csists of three spaces: young generati (YG), old generati (OG) and the permanent generati. Figure 2 In Java, objects are first allocated to the young generati space. Objects are promoted to the old generati whose age the age of an object is the number of minor GCs it has survived is more than the internally determined (by the VM) threshold. 1 VM, Java class metadata and interned Strings and class static variables are stored in the permanent generati. Input Data Figure 3 presents a bar plot of the size of GC logs for the various s (34 in total) in the cluster chosen for analysis. Note that the chosen cluster correspds to a producti cluster for a Netflix applicati. Figure 3 1 The HotSpot VM command line opti, -XX:MaxTenuringThreshold=<n>, can be used to set the threshold for tenuring objects to the old generati.

4 The GC log correspding to i- 985ceee0 has ly 8 lines as the came line close to when the data was captured. Computati of, say, standard deviati promoti size for the results in an invalid value. Hence, the is excluded from outlier detecti analysis. Data Pre- processing The format of a GC log varies from e type of GC to another. Further, given a GC type, the format of the GC log varies based the JVM flags specified at run time. The Netflix applicati chosen uses CMS (ccurrent- mark- sweep) GC. The GC Log Analysis Engine developed for this project supports various GC log formats correspding to the different JVM optis. The following shows an excerpt of a GC log from e of the s. Lines highlighted in blue correspding to GC events at different time stamps. Example GC log excerpt T16:17: : : [GC : [ParNew Desired survivor size bytes, new threshold 1 (max 3) - age 1: bytes, total - age 2: bytes, total - age 3: bytes, total : K->419392K( K), secs] K->493482K( K), secs] [Times: user=2.73 sys=0.24, real=0.80 secs] T16:17: : : [GC : [ParNew Desired survivor size bytes, new threshold 1 (max 3) - age 1: bytes, total : K->419392K( K), secs] K->922326K( K), secs] [Times: user=4.49 sys=0.38, real=1.30 secs] T16:17: : : [GC : [ParNew Desired survivor size bytes, new threshold 1 (max 3) - age 1: bytes, total : K->419392K( K), secs] K-> K( K), secs] [Times: user=5.93 sys=0.48, real=1.90 secs] T16:17: : : [GC [1 CMS-initial-mark: K( K)] K( K), secs] [Times: user=0.28 sys=0.01, real=0.29 secs] T16:17: : : [CMS-ccurrent-mark-start] T16:17: : : [CMS-ccurrent-mark: 1.985/1.997 secs] [Times: user=4.22 sys=0.15, real=1.99 secs] T16:17: : : [CMS-ccurrent-preclean-start] T16:17: : : [CMS-ccurrent-preclean: 0.099/0.100 secs] [Times: user=0.20 sys=0.00, real=0.10 secs] T16:17: : : [CMS-ccurrent-abortable-preclean-start] The numbers before and after the arrow of K->419392K indicate the size of the young generati before and after garbage collecti, respectively. After minor collectis the size includes some objects that are garbage (no lger alive) but that cannot be reclaimed. These objects are either ctained in the tenured generati, or referenced from the tenured or permanent generatis. The numbers before and after the arrow of K->493482K indicate the size of the before and after garbage collecti. References mentied in the last secti throw further light the how to read the other details of a CMS GC log. Mining GC Logs Time Series Analysis Owing to agile product development (which was e of the factors to move to the cloud from the datacenter), new applicatis and features are pushed to the cloud with high frequency. This, at times, results in introducti of performance regressis in GC performance. One of the ways to detect the time and extent of such regressi is analysis

5 of time series of the various GC metrics such as young generati over time. Multi- tenancy and hardware failure2 can lead to a degradati in performance to end- users due to service unavailability and can potentially result in losses to the business in both revenue as well as lg term reputati. Thus, detecting bad" s in a cluster, ASG in the current ctext, is key to maintain low latency and high throughput. Comparative time series analysis of GC performance across the entire cluster is e of the ways to detect bad" s. The data/plots presented in this secti were generated using R. Heap Usage Summary Heap may across s for a multitude of reass. For example, owing to n- perfect load balancer, higher number of requests may be directed to a particular that would result in higher and csequently increase GC pressure. As a first cut, we plotted the time series of for all the s to obtain a birds eye view of the entire cluster. Any obvious outlier can be detected from such a plot. Figure 4 presents the time series of of the chosen cluster Time series of for STATS Heap Usage (MB) Time Figure 4 From the time series above we note that there are no obvious outliers. Next, for each time series we compute the first and secd moment mean and standard deviati respectively. The data is presented below: 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: , standard deviati: Hoelzle and Barroso point out that hardware failure in the cloud is more of a norm than excepti. Dai et al. list the various types of failures in the cloud. Vishwanath and Nagappan reported that 8% of all servers can expect to see at least 1 hardware incident in a given year and that this number is higher for machines with lots of hard disks.

6 20: , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: : , standard deviati: For best performance, e like to minimize both the mean (μ H) and the standard deviati (σ H). In other words, e would like to maximize the following metric: 1 µμ σ The metric is similar to Sharpe Ratio (used in finance) but in the current case the mean (μ H) is the denominator as we would like to have lower mean. 3 Anomaly Detecti We computed the above metric for each and plotted a scatter plot using R. Figure 4 shows the scatter plot for the metric /(Avg Heap Us * Std Dev) Node Figure 4 Next, we used k- s to determine outlier s. We tried different values of k=2, 3 and narrowed down to k=3 as it clustered the s with low values of the metric. A scatter plot with clustering- based color coding is shown in Figure 5. 3 A high increases memory pressure which in turn adversely impacts performance.

7 10 8 /(Avg Heap Us * Std Dev) Node Figure 5 From Figure 5 we see that there are set of 11 s which have a low (relative to others) of the metric. Young Gen Usage Summary Akin to overall, we extracted the time series for Young Gen for each and computed the mean and standard deviati for each time series. The data is presented below: young gen 0: , standard deviati: young gen 1: , standard deviati: young gen 2: , standard deviati: young gen 3: , standard deviati: young gen 4: , standard deviati: young gen 5: , standard deviati: young gen 6: , standard deviati: young gen 7: , standard deviati: young gen 8: , standard deviati: young gen 9: , standard deviati: young gen 10: , standard deviati: young gen 11: , standard deviati: young gen 12: , standard deviati: young gen 13: , standard deviati: young gen 14: , standard deviati: young gen 15: , standard deviati: young gen 16: , standard deviati: young gen 17: , standard deviati: young gen 18: , standard deviati: young gen 19: , standard deviati: young gen 20: , standard deviati: young gen 21: , standard deviati: young gen 22: , standard deviati: young gen 23: , standard deviati: young gen 24: , standard deviati: young gen 25: , standard deviati:

8 young gen 26: , standard deviati: young gen 27: , standard deviati: young gen 28: , standard deviati: young gen 29: , standard deviati: young gen 30: , standard deviati: young gen 31: , standard deviati: young gen 32: , standard deviati: young gen 33: , standard deviati: Anomaly Detecti We computed the above metric for each and plotted a scatter plot using R. Figure 4 shows the scatter plot for the metric /(Avg Young Generati Use * Std Dev) Node Figure 6 Next, we used k- s to determine outlier s. Akin to anomaly detecti based, for anomaly detecti based young gen, we tried different values of k=2, 3 and narrowed down to k=3. A scatter plot with clustering- based color coding is shown in Figure 7.

9 10 4 /(Avg Young Generati Use * Std Dev) Node Figure 7 Unlike clustering for, from Figure 7 we note that k=3 is probably not the ideal choice as 32 and s 33 form a natural separate cluster. Based Figure 7, s 32 and 33 can be regarded as outliers. However, based Figure 5, 33 is not an outlier. As a first approximati, e can potentially take an intersecti of the two outlier sets (obtained based clustering of overall and young gen ). Csequently, 32 would be csidered as an outlier in the current ctext. References [1] [2] [3] [4] [5] [6]