Automated Analysis of the Results of Performance Regression Tests

Transcription

1 Automated Analysis of the Results of Performance Regression Tests

2 Zhen Ming (Jack) Jiang Assistant Professor, Department of Electrical Engineering and Computer Science, Lassonde School of Engineering, York University. Toronto, ON, Canada. Head of Software Construction, AnaLytics and Evaluation (SCALE) Lab Co-founder and co-organizer for the International Workshop on Large-Scale Testing (LT) Research Interests: Software Performance Engineering Mining Software Engineering Data Software Construction, AnaLysis and Evaluation (SCALE) Lab LT 2012 present

3 Most field problems for large scale systems are rarely functional instead they are load-related

4 Software Testing (Functional) Regression Testing Unit Testing System Integration Testing Non-Functional Testing Performance Regression Testing

5 Performance Regression Testing Mimics multiple users repeatedly performing the same tasks Take hours or even days Produces GB/TB of data that must be analyzed

6 Is the system ready for release?

7 Data Cleanup/ Abstraction Model Derivation Past Test(s) Current Test Data Cleanup/ Abstraction Test Validation

8 Studied Systems

9 Execution Logs

10 Automated Functional Analysis (E2, E3) are always together: (acquire_lock, release_lock) (open_inbox, close_inbox) If we see (E2, E6) this might be a problem E1 E2 E3 E4 E1 E2 E3 E4 E1 E2 E3 E4 E1 E2 E6 E4

11 Dell DVD Store 99.99% reduction in viewed log lines

12 Automated Performance Analysis Each load test has periods of peak/light load Fluctuations of response time Test # 1 Test # 2 Test # 3 Same workload applied across different tests Similar response time distributions

13 Performance Analysis Report - Visual Comparison Distribution Evolution

14 Performance Analysis Report - Step-wise Performance Diagnosis Stepwise We discovered a MySQL performance bug

15 Performance Counters

16 [Nguyen et al., ICPE 2012] Deriving Performance Ranges Using Control Charts Normal Operations Suspicious Test Derive control charts from the past good tests Flag new tests as anomalous if there are many violations in the control charts

17 Deriving Frequent Itemset from Past Test(s) Time DB read/sec Throughput Request Queue Size 10:00 Medium Medium Low 10:03 Medium Medium Low 10:06 Low Medium Medium 10:09 Medium Medium Low 10:12 Medium Medium Low 10:15 Medium Medium Low

18 Deriving Frequent Itemset from Past Test(s) Time DB read/sec Throughput Request Queue Size 10:00 Medium Medium Low 10:03 Medium Medium Low 10:06 Low Medium Medium 10:09 Medium Medium Low 10:12 Medium Medium Low 10:15 Medium Medium Low DB read/sec Medium Throughput Medium Request Queue size Low

19 Deriving Frequent Itemset from Past Test(s) DB read/sec Medium Throughput Medium Request Queue size Low DB read/sec Medium DB read/sec Medium Request Queue Size Low Throughput Medium Request Queue Size Low Throughput Medium Request Queue size Low Throughput Medium DB read/sec Medium

20 Automated Derivation of Performance Rules We have derived from the past test(s) a set of performance rules: DB read/sec Medium Throughput Medium Request Queue size Low Premise Consequence Flags tests where the rule does not hold DB read/sec Medium Throughput Medium Request Queue size High

21 Counter Analysis Report

22 Counter Analysis Report - Examine the Anomalous Period

23 Deriving Transaction Profiles Using Queuing Theory Every transaction visits various resources (e.g., CPU and disk) The complete series of service demands for one transaction is known as the Transaction Profile (TP) TP for transaction type A TP should be stable across different tests. Otherwise, there is an anomaly [Ghaith et al., CSMR 2013, STVR 2015]

24 Correlating Logs and Counters

25 Combining Logs and Counters 00:01, USER starts a conversation with USER 00:01, USER says MSG to USER 00:02, USER says MSG to USER 00:11, USER says MSG to USER 00:12, USER says MSG to USER 00:18, USER ends a conversation with USER 00:00, 5MB 00:08, 15MB 00:16, 15MB 00:24, 5MB

26 Memory (MB) Diagnosing Memory-Related Problems :00 00:08 00:16 00:24 Time

27 Creating A Research Community

28 Large-Scale Testing includes all different objectives and strategies of testing large-scale software systems using load. Examples of large-scale testing include live upgrade testing, load testing, high availability testing, operational profile testing, performance testing, reliability testing, stability testing and stress testing. Workshop: Contact:

29