The Theory And Practice of Testing Software Applications For Cloud Computing. Mark Grechanik University of Illinois at Chicago

Transcription

1 The Theory And Practice of Testing Software Applications For Cloud Computing Mark Grechanik University of Illinois at Chicago

2 Cloud Computing Is Everywhere Global spending on public cloud services estimated to reach $110.3Bil by Does it affect engineering software and in what way?

3 Software Engineers Are Not Sure What Cloud Computing Brings to Them

4 Testing In the Cloud OR Testing For the Cloud? Testing in the cloud means to run multiple test scripts in parallel in different virtual machines (VMs) Use cloud to speed up testing Testing for the cloud means that certain properties of the application are verified and validated to ensure that it works correctly when deployed in the cloud Test applications to ensure that they will work in the cloud

5 Is Cloud Computing Orthogonal To Software Testing?

6 Or The Cloud Will Get Ya If You Ignore It!

7 Software Engineering Requirements Gathering Design Development Tes6ng 4 Maintenance and Evolu6on 5 Cloud Compu,ng Is Not A Part of So9ware Development Lifecycle

8 What does it mean to develop and test software applications for cloud platforms?

9 Understanding Fundamentals of Cloud Computing

10 Cloud Is Not A New Thing Distributed Computing Grid Computing P2P computing Internet computing Service-oriented computing Cloud computing

11 What Is The Fundamental Nature of Cloud Computing? Traditional Computing Forecast infrastructure needs Purchase infrastructure hardware Own and maintain infrastructure Infrastructure is rented Cloud Computing Resources are dynamically (de)allocated Applications are deployed in virtual machines

12 Cloud Is A Many Splendored Thing

13 Elasticity of Cloud Computing Cloud infrastructure re-allocate resources on demand and are considered elastic Scale out and up adding resources to VMs as demand increases Scale in and down releasing resources to VMs as demand decreases Highly elastic clouds can re-provision resources rapidly and automatically in response to demand Stakeholders pay only for using, rather than for owning and employing the hardware/software infrastructures and technical staff that support their applications

14 The Dream of Elastic Cloud I wish to significantly reduce the costs by renting rather than owning elastic cloud execution infrastructure!

15 What Google Offers

16 What Amazon AWS Offers Auto Scaling enables you to closely follow the demand curve for your applications, reducing the need to provision Amazon EC2 capacity in advance. For example, you can set a condition to add new Amazon EC2 instances in increments of 3 instances to the Auto Scaling Group when the average CPU utilization of your Amazon EC2 fleet goes above 70 percent; and similarly, you can set a condition to remove Amazon EC2 instances in the same increments when CPU Utilization falls below 10 percent. Often, you may want more time to allow your fleet to stabilize before Auto Scaling adds or removes more Amazon EC2 instances. You can configure a cool-down period for your Auto Scaling Group, which tells Auto Scaling to wait for some time after taking an action before it evaluates the conditions again. Auto Scaling enables you to run your Amazon EC2 fleet at optimal utilization.

17 Google And Other Cloud Providers Offer Heuristics

18 Cost-Utilization Curve cost Lost customers Lost money A 1 O U O A 2 time

19 The Problem Application s behavior is difficult to predict Increased cost of the application deployment Cloud does not (de)allocate resources effectively Decreased performance of the application 60% or companies were not satisfied with performance and 46% cited deployment cost

20 Good Performance Is Important

21 Excellent Performance Is Even More Important!

22 Balancing Cost And Performance Resources cost money, per usage Precise cloud elasticity is an answer to the problem of providing sufficient resources to software applications while minimizing the cost of deployment

23 Connect Cloud Deployment With Performance Testing Cloud Deployment Provisioning Strategies Performance Testing

24 Cloud Computing And Software Engineering Software Engineering Lifecycle Modeling Testing Optimizing Maintaining Evolving Cloud Computing Hardware Load Balancer Virtual Machines Interfaces Provisioning Strategies Key Cloud Concepts Cost of Variability of Resources resources

25 Key Insights Key software and data artifacts are generated during different stages of the software engineering lifecycle could be, but are not currently being, used by the cloud infrastructures to improve predictive provisioning of resources. Applications running in the cloud do not provide important feedback to stakeholders about any performance problems their applications may have and that might ultimately require more resources and, thus, increase operational costs. There is no two-way flow of information between software development and cloud deployment of these applications.

26 Model of Cloud Computing VM 1 VM 5 P P P Load Balancer P P VM 4 P VM 2 P P P P VM 3

27 Provisioning Strategies

28 Provisioning Strategies

29 Performance Rules Descriptive rules for selecting test input data play a significant role in software testing, where these rules approximate the functionality of the application. Example rule: some customers will pose a high insurance risk if these customers have one or more prior insurance fraud convictions and deadbolt locks are not installed on their premises. Bottleneck

30 Putting Rules And Strategies Together For Cloud Computing Learn Performance Rules Synthesize Provisioning Strategies Use Provisioning Strategies

31 Provisioning Resources with performance Software Test automation (PRESTO) Learn Performance Rules Synthesize Provisioning Strategies Use Provisioning Strategies

32 Performance Rules Are Behavioral Models Behavioral model is a collection of workload profiles, constraints, performance counters, and various relations among components of the application. Performance rules are examples of applications behavioral models. Our goal is to obtain and use these models to synthesize provisioning strategies

33 Finding Rules That Describe Performance Problems A goal is to find situations when applications unexpectedly exhibit worsened characteristics for certain combinations of input values.

34 Finding This Situation Is Like Getting a Perfect Hand!

35 Example: Bottlenecks A slow component of the application A C B Bottlenecks or hot spots are phenomena where the performance of the application is limited by one or few components

36 Bottlenecks in Nontrivial Applications Inputs Output

37 Performance Testing Applications Client Client calls Server methods Server Server interacts with back-end data storages Data base The Client executes methods of the Server using different combination of input parameter values

38 Performance Testing Applications Client Client receives data from Server Server Server receives data from Database Data base Performance can be measured in the number of roundtrips per time unit

39 Automating Performance Testing Applications Client Client calls Server methods and receives data Server Server interacts with back-end data storages Data base Automated test scripts simulate clients who perform transactions against the Server, and they use input data from the backend database

40 Automating Performance Testing Applications Server Server interacts with back-end data storages Data base Automated test scripts simulate clients who perform transactions against the Server, and they use input data from the backend database

41 A Problem With Performance Testing of Large Applications A medium-size renters insurance application has a large universe of input data. For 78,000,000 customer profiles it would take close to 1,500 years to test this application on all profile inputs provided that it takes only 10mins per input. Just touching one customer profile entry for 1sec, it takes about 2.5 years to touch them all! Some customer profiles may lead to significant load on different resources.

42 A Problem With Performance Testing of Large Applications How to select a manageable subset of the input data without compromising the effectiveness of performance testing. We need an insight into selecting this subset!

43 An Example Most of the renters are good customers, have hardly any accident/claims or fraudulent activities. That s how insurance companies make money. Unfortunately, the data of these good customers are not always good for performance testing purpose. Randomly selecting customer profiles does not work well keep encountering good customers. Yet, exploratory performance testing is one of the main ways of performance testing applications in industry!

44 The Intuition Behind Performance Testing Trigger more computation More data to process in programs Bigger executing footprint Bigger database footprint Trigger more database queries

45 Core Idea Select only these customer inputs that trigger intensive computations. These computations most likely involve significant interactions with databases.

46 Performance Testing Is Laborious and Difficult And It Is Not Designed For The Cloud

47 A Computing Primitive in PRESTO - FOREPOST Feedback-ORiEnted PerfOrmance Software Testing (FOREPOST) finds performance problems automatically by learning and using rules that describe classes of input data that lead to intensive computations. FOREPOST is an adaptive, feedback-directed learning testing system that learns rules from execution traces and uses these learned rules to select test input data automatically to find more performance problems in applications when compared to exploratory random performance testing.

48 Compute a Function That Represents the Application FOREPOST CLOUD f: Input -> Performance

49 Components of FOREPOST Adap6ve Test Script Run6me Monitoring Machine Learning Informa6on Compression 4 BoPleneck Detec6on Algorithm 5 Adap6ve test script uses rules that are issued by machine learning components to select input data that lead to more intensive computa6ons. When the test script executes the applica6on, its execu6on traces are collected as part of run6me monitoring, leading to (re)learning rules.

50 Some Statistics It takes about 10mins to execute the instrumented renters app end-2-end automatically using a test script. An average run invokes over 3,000 methods more than 3,000,000 times and it retrieves over 1Mb of data. An average execution traces contains over 1Gb of data. It takes less than 10mins on average to process and analyze one trace.

51 Constructing Data For a Learner and Learning Rules Input_1_Name Input_2_Name Input_k_Name Class Value_p Value_q Value_m Good Value_p Value_q Value_m Bad Value_p Value_q Value_m Good

52 Finding Bottlenecks A slow component of the application A C B Bottlenecks or hot spots are phenomena where the performance of the application is limited by one or few components

53 Finding Bottlenecks Cluster Execution Traces Rank Methods by Their Significance In Each Cluster Compute Logical Difference Between Clusters Good Bad Good Bad

54 FOREPOST Architecture Profiler Test Script Rules Learner Good test traces Method and data statistics ICA Advisor Bad test traces Method weights Execution Trace Analyzer Trace Statistics Trace Clustering

55 Result for Renters App

56 Result for JPetStore

57 PRESTO Architecture Customers 16 1 Auto Test Scripts 13 Load Balancer Resource Controller App Load Model 6 3 FOREPOST 9 Perturbation Operators Provisioning Strategies Performance Rules

58 Experimenting With CPU Bottlenecks Using CloudStack Throughput (number of URLs per second) Random Basic Δ1 Δ2 Δ3 Δ4 Δ5 0 JPetStore - 5 users JPetStore - 15 users JPetStore - 30 users Dell DVD Store - 5 users Dell DVD Store - Dell DVD Store - 15 users 30 users

59 Experimenting With Database Bottlenecks Using CloudStack Throughput (number of URLs per second) Random Basic Δ1 Δ2 Δ3 Δ4 Δ5 0 JPetStore - 5 users JPetStore - 15 users JPetStore - 30 users Dell DVD Store - 5 users Dell DVD Store - Dell DVD Store - 15 users 30 users

60 Conclusions This proposed research program is novel, as to the best of our knowledge, there exists no work that utilizes automatic feedback-directed adaptive test scripts to find improve cloud elasticity. The results of this work show that the proposed approach can effectively locate performance problems and guide stakeholders to improve the quality of software that is deployed in the cloud.

61