Evaluating the Prediction Accuracy of Generated Performance Models in Up- and Downscaling Scenarios

Transcription

1 Stuttgart, Germany, Evaluating the Prediction Accuracy of Generated Performance Models in Up- and Downscaling Scenarios Symposium on Software Performance (SOSP) 214 Andreas Brunnert 1, Stefan Neubig 1, Helmut Krcmar 2 1 fortiss GmbH, 2 Technische Universität München fortiss GmbH An-Institut Technische Universität München

2 Agenda Motivation and Vision Performance Model Generation Data Collection Data Aggregation Model Generation Evaluation SPECjEnterprise21 Overhead Evaluation Experiment Setup Scenario Description Scenario Results Upscaling Downscaling Future Work 2

3 Agenda Motivation and Vision Performance Model Generation Data Collection Data Aggregation Model Generation Evaluation SPECjEnterprise21 Overhead Evaluation Experiment Setup Scenario Description Scenario Results Upscaling Downscaling Future Work 3

4 Motivation & Vision Numerous performance modeling approaches are available to evaluate the performance (i.e., response time, throughput, resource utilization) of enterprise applications Performance models are especially useful for scenarios that cannot be tested on real systems, e.g.: Scaling a system up or down in terms of the available hardware resources (e.g., number of CPU cores) during the capacity planning process. Creating a performance model requires considerable manual effort low adoption rates of performance models in practice 4

5 Motivation & Vision To increase the adoption rates of performance models in practice! For that purpose, we have proposed an automatic performance model generation approach for Java Enterprise Edition (EE) applications. This work improves the existing approach by further reducing the effort and time for the model generation.... This work evaluates the prediction accuracy of these generated performance models in up- and downscaling scenarios, i.e.: Increased and reduced number of CPU cores Increased and reduced number of concurrent users 5

6 Agenda Motivation and Vision Performance Model Generation Data Collection Data Aggregation Model Generation Evaluation SPECjEnterprise21 Overhead Evaluation Experiment Setup Scenario Description Scenario Results Upscaling Downscaling Future Work 6

7 Performance Model Generator Monitoring Data Persistence Service Java EE Application Performance Model Generation Overview PMWT Agent CSV CSV CSV PMWT Connector MBeans Monitoring Database Performance Model 1. Data Collection 2. Data Aggregation 3. Model Generation (adapted from Willnecker et al. (214)) 7

8 Performance Model Generator Monitoring Data Persistence Service Java EE Application Performance Model Generation Data Collection PMWT Agent CSV CSV CSV MBeans Monitoring Database PMWT Connector Performance Model Java EE Application Web Tier Business Tier Servlet Filters Web Components (Servlets/JSPs) EJB Interceptors Enterprise JavaBeans (EJBs) JDBC Wrappers Enterprise Information Systems Tier Data collected: EJB and Web components EJB and Web component operations EJB and Web component relationships on the level of single component operations Resource demands for single component operations (CPU, Memory) 8

9 Performance Model Generator Monitoring Data Persistence Service Java EE Application Performance Model Generation Data Aggregation PMWT Agent CSV CSV CSV MBeans Monitoring Database PMWT Connector Performance Model BranchMetrics -invocationcount : long(idl) -totalcpudemand : long(idl) -totalresponsetime : long(idl) -totalallocatedheapbytes : long(idl) 1 1 BranchDescriptor * 1 ParentOperationBranch * 1 JavaEEComponentOperationMBean OperationCallLoopCount -loopcount : long(idl) -loopcountoccurrences : long(idl) * 1 1 * {ordered} ExternalOperationCall * 1 * OperationIdentifier -type : string(idl) -componentname : string(idl) -operationname : string(idl) (Brunnert et al. 214_1) 9

10 Performance Model Generator Monitoring Data Persistence Service Java EE Application Performance Model Generation Model Generation PMWT Agent CSV CSV CSV MBeans Monitoring Database PMWT Connector Performance Model Except for the usage model, default models for all model layers of the Palladio Component Model (PCM) are generated automatically: Usage Model Repository Model Repository model containing the components of an EA, their relationships and resource demands System model containing the deployment units detected during the data collection (no single components) System Model Allocation Model A simple resource environment with one server and an allocation model that maps all deployment units to this server Resource Environment (adapted from a presentation for Brunnert et al. (214_2) 11

11 Agenda Motivation and Vision Performance Model Generation Data Collection Data Aggregation Model Generation Evaluation SPECjEnterprise21 Overhead Evaluation Experiment Setup Scenario Description Scenario Results Upscaling Downscaling Future Work 12

12 Evaluation SPECjEnterprise21 Benchmark Driver / Emulator Simulates interactions with the SUT Defines workload Automobile Manufacturer Orders Domain (CRM) Manufacturing Domain Supplier Domain (SCM) Database Server SPECjEnterprise 21 Architecture (SPEC, 29) 13

13 Evaluation SPECjEnterprise21 (Standard Performance Evaluation Corporation 29) Workload defined by driver Business Transactions Browse,, Predefined sequence of HTTP requests including probabilities SPECjEnterprise 21 Architecture (SPEC, 29) Orders Domain Architecture (Brunnert et al. 213) 14

14 Evaluation Overhead Evaluation 1/2 Monitoring code is called before and after each monitored invocation Considerable instrumentation overhead! Overhead Evaluation: CPU & Heap vs. CPU only 4 CPU cores 2 GB RAM 6 Users Only steady state data is collected 15

15 Evaluation Overhead Evaluation 2/2 CPU & Heap CPU only Component Operation All levels Top level All levels Top level CPU Heap CPU Heap CPU CPU 1 app.sellinventory 1.23 ms 33,65 B 3.1 ms 225,39 B.756 ms 3.3 ms 2 CustomerSession.sellInventory.785 ms 6,45 B ms - 3 CustomerSession.getInventories.594 ms 49,54 B ms - 4 OrderSession.getOpenOrders.954 ms 7,6 B ms - 5 dealerinventory.jsp.sellinventory.18 ms 16,66 B ms - Total Resource Demand ms 23,9 B 3.1 ms 225,39 B 3.15 ms 3.3 ms Mean Data Collection Overhead.116 ms 1378 B.3 ms (Brunnert et al. 214_1) 16

16 Evaluation Overhead Evaluation 2/2 CPU & Heap CPU only Component Operation All levels Top level All levels Top level CPU Heap CPU Heap CPU CPU 1 app.sellinventory 1.23 ms 33,65 B 3.1 ms 225,39 B.756 ms 3.3 ms 2 CustomerSession.sellInventory.785 ms 6,45 B ms - 3 CustomerSession.getInventories.594 ms 49,54 B ms - 4 OrderSession.getOpenOrders.954 ms 7,6 B ms - 5 dealerinventory.jsp.sellinventory.18 ms 16,66 B ms - Total Resource Demand ms 23,9 B 3.1 ms 225,39 B 3.15 ms 3.3 ms Mean Data Collection Overhead.116 ms 1378 B.3 ms (Brunnert et al. 214_1) 17

17 Evaluation Overhead Evaluation 2/2 CPU & Heap CPU only Component Operation All levels Top level All levels Top level CPU Heap CPU Heap CPU CPU 1 app.sellinventory 1.23 ms 33,65 B 3.1 ms 225,39 B.756 ms 3.3 ms 2 CustomerSession.sellInventory.785 ms 6,45 B ms - 3 CustomerSession.getInventories.594 ms 49,54 B ms - 4 OrderSession.getOpenOrders.954 ms 7,6 B ms - 5 dealerinventory.jsp.sellinventory.18 ms 16,66 B ms - Total Resource Demand ms 23,9 B 3.1 ms 225,39 B 3.15 ms 3.3 ms Mean Data Collection Overhead.116 ms 1378 B.3 ms Without heap monitoring, the Mean Data Collection Overhead drops dramatically! We focus on collecting CPU demand only! (Brunnert et al. 214_1) 18

18 Evaluation Experiment Setup Model generation Collect only CPU demands using Servlet Filters and EJB Interceptors Only steady state data is collected SUT under moderate load (~ 4-6 %) Model simulation Generated PCM models are taken as input for SimuCom Results compared with benchmark runs Benchmark runs Servlet Filter at front controller to measure response times and throughput JMX connection to measure CPU utilization of JVM Only steady state data is collected Performed three times (same weight) 19

19 Evaluation Scenario Description Upscaling: How many resources do I need for an increasing workload? Workload: 6 Users 9 Users 12 Users CPU: 4 Cores 6 Cores 8 Cores Model generation: 4 cores (~52 % load) Downscaling: How many resources do I need for the given workload? Workload: 8 Users CPU: 8 Cores 6 Cores 4 Cores Model generation: 8 cores (~39 % load) 2

20 Evaluation Scenario Results: Upscaling 1/2 CPU Cores Number of Users Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error Browse ms 65.6 ms 2.91 % % ms ms % % % % 3.88 % 8.27 ms 9.73 ms % % Browse ms ms % % ms ms 4.69 % % % % 9.42 % 8.95 ms 8.72 ms 2.6 % % Browse ms ms % % ms ms % % % % 18.9 % 9.31 ms 8.46 ms 9.15 % % (Brunnert et al. 214_1) 21

21 Evaluation Scenario Results: Upscaling 1/2 CPU Cores Number of Users Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error Browse ms 65.6 ms 2.91 % % ms ms % % % % 3.88 % 8.27 ms 9.73 ms % % Browse ms ms % % ms ms 4.69 % % % % 9.42 % 8.95 ms 8.72 ms 2.6 % % Browse ms ms % % ms ms % % % % 18.9 % 9.31 ms 8.46 ms 9.15 % % (Brunnert et al. 214_1) 22

22 Evaluation 3 Scenario Results: Upscaling / CPU cores, 6 users 6 CPU cores, 9 users 8 CPU cores, 12 users Upscaling + Upscaling Upscaling +2 Upscaling Upscaling pmw.fortiss.org SOSP 214, Stuttgart, Germany, Upscaling Upscaling +4 Upscaling (Brunnert et al. 214_1)

23 Evaluation Scenario Results: Downscaling 1/2 CPU Cores Number of Users Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error Browse ms 64.3 ms 1.5 % % ms ms 2.49 % % % % 5.99 % 9.36 ms 9.33 ms.25 % % Browse ms 66.3 ms 2.35 % % ms 13.8 ms 4.45 % % % % 1.21 % 9.5 ms 9.64 ms 6.57 % % Browse ms ms % % ms 17.4 ms % % 65.6 % 7.27 % 7.12 % 8.93 ms ms % % (Brunnert et al. 214_1) 24

24 Evaluation Scenario Results: Downscaling 1/2 CPU Cores Number of Users Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error Browse ms 64.3 ms 1.5 % % ms ms 2.49 % % % % 5.99 % 9.36 ms 9.33 ms.25 % % Browse ms 66.3 ms 2.35 % % ms 13.8 ms 4.45 % % % % 1.21 % 9.5 ms 9.64 ms 6.57 % % Browse ms ms % % ms 17.4 ms % % 65.6 % 7.27 % 7.12 % 8.93 ms ms % % (Brunnert et al. 214_1) 25

25 Evaluation Downscaling - Scenario Results: Downscaling /2 8 CPU cores, 8 users Downscaling Downscaling -2 Downscaling (Brunnert et al. 214_1) CPU cores, 8 users Downscaling Downscaling -4 Downscaling Downscaling pmw.fortiss.org SOSP 214, Stuttgart, Germany, CPU cores, 8 users

26 Agenda Motivation and Vision Performance Model Generation Data Collection Data Aggregation Model Generation Evaluation SPECjEnterprise21 Overhead Evaluation Experiment Setup Scenario Description Scenario Results Upscaling Downscaling Future Work 27

27 Future Work Usage model generation Master Thesis recently finished based on session traces Christian Vögele currently extends this approach Representing distributed systems Distributing the transaction ID within distributed Java EE environments using the Application Response Measurement (ARM) standard or similar means Aggregating data from multiple Java EE instances Recently finished a prototype for distributed deployments communicating over SOAP Setting thread limits for each tier (Web, EJB, JDBC Conn. Pools) Supporting new component types: JSF, web-services, Message driven beans, Other Java EE 7. enhancements, e.g. Batch Jobs,.. 28

28 Performance Model Generator Monitoring Data Persistence Service Java EE Application Future Work Using APM data for the model generation (collaboration with Dynatrace) PMWT Agent CSV CSV CSV PMWT Connector MBeans Monitoring Database Performance Model dynatrace Agent dynatrace Server dynatrace Connector Session Store dynatrace Performance Warehouse (Willnecker et al. 214) 29

29 Bibliography Brunnert, A.; Vögele, C.; Krcmar, H. (213): Automatic performance model generation for java enterprise edition (ee) applications, in: M. S. Balsamo, W. J. Knottenbelt, A. Marin (Eds.), Computer Performance Engineering, Vol of Lecture Notes in Computer Science, Springer Berlin Heidelberg, 213, pp Brunnert, A.; Neubig, S.; Krcmar, H. (214): Evaluating the prediction accuracy of generated performance models in up- and downscaling scenarios, in: Proceedings of the Symposium on Software Performance, SOSP 14, 214, pp Brunnert, A.; Wischer, K.; Krcmar, H. (214): Using architecture-level performance models as resource profiles for enterprise applications, in: Proceedings of the 1th International ACM Sigsoft Conference on Quality of Software Architectures, QoSA '14, ACM, New York, NY, USA, 214, pp Standard Performance Evaluation Corporation (SPEC) (29): SPECjEnterprise21 Design Document. accessed at Willnecker, F.; Brunnert, A.; Gottesheim, W.; Krcmar, H. (214): Using dynatrace monitoring data for generating performance models of Java EE applications. Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, Austin, TX, USA, (accepted for publication, to appear). 3

30 Q&A Andreas Brunnert, Stefan Neubig pmw.fortiss.org 31