Automatic Extraction of Probabilistic Workload Specifications for Load Testing Session-Based Application Systems

Transcription

1 Bratislava, Slovakia, Automatic Extraction of Probabilistic Workload Specifications for Load Testing Session-Based Application Systems André van Hoorn, Christian Vögele Eike Schulz, Wilhelm Hasselbring, Helmut Krcmar

2 Problem Statement Situation Workload specification and execution essential to evaluate performance properties of session-based application systems (e.g., Krishnamurthy et al. 2006, Menascé et al. 1999, Arlitt et al. 2001) Complication Manual creation of representative workload specifications is difficult, time consuming, and error-prone (Shams et al. 2006) Extraction and specification of workloads strongly depends on the used workload generation tool Throughput Workload Specification Workload Generator SUT Performance Monitor Workload Intensity 2 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

3 Problem Statement Situation Workload specification and execution essential to evaluate performance properties of session-based application systems (e.g., Krishnamurthy et al. 2006, Menascé et al. 1999, Arlitt et al. 2001) Complication Manual creation of representative workload specifications is difficult, time consuming, and error-prone (Shams et al. 2006) Extraction and specification of workloads strongly depends on the used workload generation tool Resolution: WESSBAS 1) System- and tool-agnostic modeling of probabilistic workloads of session-based application systems 2) Automatic extraction of these specifications from running systems including the clustering of navigational patterns 3) Transformation of these specifications into load test scripts 4) Tool support for this approach 3 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

4 WESSBAS Approach 4 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

5 Related Work Extraction/characterization of workload specifications for session-based applications systems (Arlitt et al. 2001, Krishnamurthy et al. 2006) Extraction of Customer Behavior Model Graphs (CBMGs) from HTTP server logs, including K-means clustering to identify CBMGs for similar types of users (Menascé et al. 1999) A DSL-based definition of variable and dynamic load profiles and workload scenarios over time (e.g., Kistowski et al. 2014) Intermediate languages to reduce the complexity of generating different kinds of performance models (Ciancone et al. 2011, Smith et al. 2004, Woodside et al. 2005) 5 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

6 Agenda Problem Statement and Overview of Approach Related Work WESSBAS Approach 1. WESSBAS-DSL 2. Extraction of WESSBAS-DSL Instances 3. Clustering of Customer Groups 4. Transformations into Apache JMeter Evaluation Future Work 6 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

7 Background Markov4JMeter Example of an Application Model [van Hoorn et al. 2008] 7 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

8 Background Markov4JMeter Example of an Application Model [van Hoorn et al. 2008] 8 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

9 WESSBAS-DSL Application Model 9 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

10 Behavior Model + Behavior Mix Background Markov4JMeter [van Hoorn et al. 2008] 10 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

11 WESSBAS-DSL Behavior Mix and Behavior Model(s) 11 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

12 Behavior Model Extraction 12 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

13 Clustering of Customer Groups X-means (Pelleg, Moore 2000) (Absolute) Transition count matrix per session Determine number of clusters and corresponding centroids Assigns instances to the nearest cluster centroid using Euclidean distance (Non-normalized and normalized) yes Instances move group? no stop Behavior Model(s) Behavior Mix Probabilities: Each centroid represent the (relative) transition count matrix Think Times: Calculated based on resulting clusters (not part of clustering) Proportion of the number of instances per cluster Grouping based on minimum Euclidean distance and recalculation of centroids 13 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

14 Transformation into Apache JMeter Test Plans WESSBAS- DSL WESSBAS 14 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

15 Transformation into Apache JMeter Test Plans 15 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

16 Agenda Problem Statement and Overview of Approach Related Work WESSBAS Approach 1. WESSBAS-DSL 2. Extraction of WESSBAS-DSL Instances 3. Clustering of Customer Groups 4. Transformations into Apache JMeter Evaluation Future Work 16 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

17 Evaluation Goals and Methodology Research Questions 1. How accurately do the clustering results match the input Behavior Mix? 2. What is the impact of the clustering results on the workload characteristics? Methodology Instrumentation of SPECjEnterprise2010 using Kieker to obtain session logs Extraction of behavior models and behavior mix (includes clustering) Extraction of WESSBAS-DSL instances Transformation to JMeter test plans Generation of basic application model (only session layer) No input data, no guards and actions Generation of dummy HTTP requests, e.g., Create dummy web application with actionservlet Execute workload on dummy web application and measure workload with Kieker 17 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

18 SPECjEnterprise2010 Transactions Adapt SPECj dealer driver: Login / logout at beginning / end of each transaction Transaction Mix = Behavior Mix 18 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

19 Accuracy of Clustering Research Question 1 TM: Transaction Mix T: Transaction C N : Assigned Cluster MC: Percentage of misclassified N: Number of instances ED: Euclidean Distance NED: Normalized Euclidean Distance 19 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

20 Accuracy of Extracted Workload Specifications Research Question 2 Session-based metrics Request-based metrics Session length Relative counts (common to all workloads) Statistics about session-based metrics Absolute and relative counts 20 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

21 Future Work Extension for model-based performance evaluation (Vögele et. 2014) Automatic generation of application model Executable load tests Automatic learning of guards and actions Generation of protocol layer Extraction and generation of input data Support for workload intensity LIMBO (Kistowski et al. 2014) Additional transformation to alternative workload generators Online clustering to detect evolution of behavior mix Industrial case study Supplementary material (software, (meta-)models, data, scripts) publicly available online: 21 van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems

22 References M. F. Arlitt, D. Krishnamurthy, and J. Rolia. Characterizing the scalability of a large web-based shopping system. ACM TOIT, 1(1):44-69, A. van Hoorn, M. Rohr, and W. Hasselbring. Generating probabilistic and intensity-varying workload for Web-based software systems. In Proc. SIPEW '08, pages , D. Krishnamurthy, J. A. Rolia, and S. Majumdar. A synthetic workload generation technique for stress testing sessionbased systems. IEEE TSE, 32(11): , D. A. Menascé, V. A. F. Almeida, R. Fonseca, and M. A. Mendes. A methodology for workload characterization of e- commerce sites. In Proc. EC '99, pages , D. Pelleg, A. W. Moore. X-means: Extending K-means with ecient estimation of the number of clusters. In ICML '00, pages , J. v. Kistowski, N. R. Herbst, and S. Kounev. Modeling variations in load intensity over time. In Proc. LT '14, pages 1-4. ACM, M. Shams, D. Krishnamurthy, and B. Far. A model-based approach for testing the performance of web applications. In Proc. SOQUA 2006, pages ACM, C. Vögele, A. van Hoorn, H. Krcmar. Automatic Extraction of Session-Based Workload Specifications for Architecture- Level Performance Models. In Proc. LT@ICPE 2015 (to appear). ACM A. Ciancone, A. Filieri, M. Drago, R. Mirandola, and V. Grassi. Klapersuite: An integrated model-driven environment for reliability and performance analysis of component-based systems. In Objects, Models, Components, Patterns, volume 6705 of LNCS, pages Springer Berlin Heidelberg, M. Woodside, D. C. Petriu, D. B. Petriu, H. Shen, T. Israr, and J. Merseguer. Performance by unified model analysis (puma). In Proc. WOSP '05, pages 1-12, New York, NY, USA, ACM. C. Smith and C. Llado. Performance model interchange format (pmif 2.0): Xml defintition and implementation. In Proc. QEST '04, pages 38-47, Sept van Hoorn, Vögele, Schulz, Hasselbring, Krcmar WESSBAS: Workload Extraction and Specification for Session-Based Application Systems