Petri Net Modeling of Application Server Performance for Web Services

Transcription

1 Petri Net Modeling of Application Server Performance for Web Services M. Rahmani, A. Azadmanesh, H. Siy College of Information Science & Technology University of Nebraska-Omaha Omaha, NE Tel: +1(42) {crahmani, azad, Abstract A study of failure rates of a web service that is deployed in a service-oriented architecture is presented. The study focuses on the HTTP requests that are rejected by the application server. The rejections may be caused by system overloading or mismanagement of configuration parameters. An analytical model and a Stochastic Activity Network (SAN) model are developed to predict the number of such failures. The performance of the models is compared against the experimental results in a LAN environment that is used as a test-bed. The models utilize the parameters extracted from the empirical testing such as the average response time and arrival rate of the web service requests. The accuracy of the SAN model suggests that the model can be beneficial to predict the rejection rate of web services and to better understand the application server performance for those cases that are difficult to replicate in a field study. Keywords- Application Server; Simulation Model; Service Oriented Architecture; Web Service; Petri Net; Load Testing I. INTRODUCTION AND RELATED WORK Web services, such as electronic shopping, online auction, and banking services, have permeated our daily lives due to the ease of use, flexibility and reduction of cost in providing the services [1]. They are generically seen by many as applications that can be accessed over the Internet. A more precise definition of web services, provided by the World Wide Web Consortium (W3C) [2], defines it as a software system designed to support interoperable machine-to-machine interaction over a network. Web services are generally enabled through an application server software that communicates with a web server and a database server. With the prevalence of web services in real-time and critical software domains, high level of reliability and performance are expected from them. As such, a number of approaches to reliability prediction and performance evaluations have been attempted. These approaches are often not new and have been applied to reliability and performance analysis of component-based software systems. Some of the most common approaches are based on Markov Chains, Tree-based, and Petri net models [3,4,5,6,7]. The majority of research works is theoretical with less emphasis on experimental analysis to support the theoretical results [4,8,9,1,11]. For example, Zhong and Qi [8] propose a Petri net model of performance and dependability of web service composition. However, the paper emphasizes on theoretical aspect of research and does not include any experimental analysis. In [9], authors presented a simulation-based model for performance analysis and evaluation of application servers using Petri nets. However, the numerical illustration of the presented simulation model is based on hypothetical input and not real experiments. Some of the studies that use a combination of theoretical and experimental analyses include [12,13,14,15,16]. Xiao and Dohi [12] focused on the relationship between the Apache server error rate and system s performance and developed a probability model to describe this relationship. In [13], authors focused on a performance evaluation model of a web server system, which is based on a queuing model and evaluated the effectiveness of the model through the experiments in the lab environment. In [14], the authors proposed another queuing model to evaluate the performance parameters of a web server such as the response time and the blocking probability. The authors in [15] evaluate the response time and throughput of web services by collecting and analyzing large sets of data from geographically distributed locations. Goseva et al. [16] presented an empirical as well as theoretical study of architecture-based software reliability on a large open source application with 35, lines of C code. They emphasized on theoretical and experimental results on a large scale field study to test and analyze the architecture-based software reliability. This paper is focused on the performance modeling of the JBoss application server [17] with respect to response time, throughput, and rejection (blocking) rate of web service requests. The study 1) introduces an analytical method to measure the rejection rate, 2) establishes an experimental environment to collect data by generating controlled loads of web service requests to JBoss, and 3) injects the experimental results into a Petri net model that simulates the JBoss performance behavior. In general, the main contribution of the paper is the measurement and comparison of web service rejection rate from analytical, empirical and simulation (Petri net) points of view. Other than the experimental and theoretical approaches, this study is different from many other relevant research works in that the focus is geared toward understanding the effect of configuration parameters such as the maximum HTTP thread pool and the queue size in the application server. Petri nets are chosen for this study because of the

2 flexibility they provide to model systems with multithreaded executions. In comparison, analytical approaches such as Markov models are difficult to express multithreaded systems, at least because of the state-space explosion problem, as each Petri net marking becomes a single state in the Markov model. As Markov models can be seen as a finite state machine, they are more suitable for single-threaded systems. The paper is organized as follows. Section II presents a) the description of experiments, b) an analytical model, and c) the Petri net model for HTTP request rejection rate in JBoss. Section III focuses on experimental and analytical results. Section IV offers some possible directions to the future research and finally section V contains some concluding remarks. II. DESCRIPTION OF EXPERIMENT A. Experimental Environment The experimental environment consists of two hosts (client and server) remotely located from each other in a LAN environment. The host server on which JBoss is running is excluded from running other tasks to ensure the consistency of data sets collected. The Duke s Bank application [18] is transformed into a web service and deployed on JBoss. JBoss uses the Apache Tomcat as its default web server. The client host generates service requests to JBoss. The bandwidth of the LAN is shared with other users not relevant to this experiment. Tools like Wireshark [19] and Ping are used to measure the round trip delay (RTD), excluding the time spent in the hosts. In comparison to the time spent in the server, the RTD is observed to be so negligible that it is ignored in the experimental analyses. The system structure is illustrated in Fig. 1. WS client SoapUI SOAP- Http SOAP- Http Response Figure 1. Client The experimental setup. JBoss AS Banking WS Server OS Windows 7 Windows Vista Processor 1.67 GHz 2.66 GHz RAM 1. GB 2. GB Software SoapUI (3.6.1) JBoss AS (4.2.2) SoapUI [2] is used to generate controlled service request loads to the server. SoapUI is an open source testing tool solution for service-oriented architectures. With a graphical user interface, SoapUI is capable of generating and mocking service requests in SOAP format. There are two main parameters in SoapUI load testing tool that can be set to control the workload of the application server: number of threads representing the virtual users, and the number of requests (runs) generated per thread. For example, if the number of threads is set to 2 and number of runs per thread is set to 1, then there are 2 clients, each sending 1 SOAP requests for a total of 2 requests. SoapUI measures several performance parameters such as the average response time, avg, transactions per second, tps, and the number of transaction requests failed in the process, which is denoted as err. Avg is the average time difference between the times a request is sent out until the response is received. Tps, also called arrival rate, is the average number of requests generated by the clients per second. The experiments have been conducted with sixteen different load configurations. For each load test, SoapUI returns the values for avg, tps, and err. Each test is repeated 5 times and the average of the returned values are calculated. The total time T for each load test configuration is computed as follows: where cnt is the total number of requests for each test. Consequently, the error and success rates for each load are computed by: The error reports generated by SoapUI consist of all types of potential errors that are generated by the network, application server, and web service itself. However, this study considers only the errors that are generated by JBoss when the HTTP requests are rejected. Therefore, is treated as. B. The Analytical Model The actual rate of requests rejected can be obtained from (2). This rate can be estimated in a different way. Recall that avg is the average response time for one request. Therefore, the service rate is 1/avg. In order for JBoss not to reject any incoming request, it should be able to use enough threads to keep up with the arrival rate. Thus, JBoss will reject no requests if the following holds: 1 threads avg tps 1 threads avg tps Otherwise some requests will be rejected. Thus, number of requests rejected per unit of time, i.e., will be: Consequently, the total number of requests rejected is: ) (4)

3 Average response time Rejection rate The accuracy of (4) depends on threads, which will be referred to as threshold. This is because any value higher than the threshold value underestimates and any value lower than the threshold overestimates the number of rejections. Thus, the following will be used instead of (4). ( ) (5) Threshold is computed from the experimental analyses. Theoretically, when the configuration parameters maxthread and acceptcount in server.xml file are set to 25 and 1 respectively, it is expected that JBoss handle 35 requests (25 in the thread pool and 1 in the queuing system). However, in real world experiments, there are many factors such as memory, processor and type of operating system that may affect the actual number of threads that can be devoted to requests. More importantly, the response time evaluated by SoapUI includes the time spent in the queuing system and the time actually spent on servicing the request. With this experiment setup in the lab and by running different tests, threshold is estimated to be around 315. This is the threshold number of threads in JBoss that leads to performance results from the test cases performed on SoapUI. It is expected that the response time, avg, to increase as the number of requests increases. Fig. 2 shows the avg numbers for different virtual users in the real experiments, where each user sends 1 requests. The figure exhibits that the response time increases up to a point, levels-off for a moderate range of requests, and then increases again Number of virtual users Figure 2. Average response time based on number of virtual users. The reduction in response time beyond 37 up to around 5 users is counterintuitive because as there are more requests the response time ought to increase instead. During this period, Fig. 3 indicates that the rejection rate is increased. Toward the end, Fig. 2 further shows sharp increase in the average response time while the rejection rate on average in Fig. 3 stays the same. This paradoxical behavior has two reasons. The first reason is that SoapUI uses the total requests in calculating the average response time, regardless of whether a request is successful or rejected. It is for this reason that the average response time of in Fig. 2 can be interpreted as the true response time because at about 37 users the number of rejections is low or almost nonexistent. The second reason is contributed to JBoss, which can be explained as follows. From 37 to around 5 users, the server is using the resources that are already setup and activated, so the system can reallocate them to other requests, such as the thread resources. Beyond 5 users, the server overhead, such as the time taken to reject the requests, accumulates as the rejection rate increases. Since SoapUI includes the rejection count in the evaluation of average response time, one way to find a good estimate of the actual response time is to use a packet capture tool such as Wireshark and evaluate the response time for each successful request from the time the request is received by the server until the response is sent back to SoapUI. This requires filtering the blocked requests and evaluating the response time for each individual successful request, which seems to be infeasible. The other approach is to use as the estimate of the actual response time for loads more than 37, where the rejections really start. This value shows the peak of response time when the system utilizes all the resources for high level loads without being biased by the rejected requests. Therefore, in this study avg = is used as the closest approximation of average response time for high level loads Number of virtual users Figure 3. rejection rate based on number of virtual users. C. The Stochastic Activity Network Model Petri nets are a graphical model for the formal specification and analysis of concurrent and discrete-event systems. Since the introduction of classic Petri nets, a number of variants have been introduced. Stochastic Petri Nets (SPN) is a subsidiary of timed Petri nets that adds non-deterministic time through randomness of transitions. Generalized Stochastic Petri Nets (GSPN) is a SPN performance analysis tool that uses the exponential random distribution, and thus conversion to Markov Chains is automated. Stochastic Activity Network (SAN) [21] is a structurally extended version of GSPN with many features such as the ability of creating complex enabling functions, probabilistic choices upon completion of activities, and reward functions. In the world of SANs, transitions (actions) are referred to as activities, which can be of two kinds: timed and instantaneous. The firing (activation) time of timed activities are exponentially distributed. Once enabled, instantaneous activities

4 complete in zero-time, and thus have priority over timed activities. Reward functions are used to measure performance or dependability related issues. The SAN model describing the behavior of the web service system is constructed using Mobius [22]. Mobius can solve SAN models, either mathematically or by simulation. Because of the types of reward rates used we have found it easier and less time consuming to work with the simulation solver. Fig. 4 provides a SAN model for the service requests that arrive at the server (JBoss) side. Recall that Duke s Bank web service is used in this experiment. This web service receives customer ID and returns all account numbers of the customer. Tarrival OG Start TJW Response Webservicedown Blockedrequest Threadpool Figure 4. The Petri net model of JBoss serving the requests. In the figure, the timed and instantaneous activities are shown by thick and thin bars, respectively. Each flat dot represents a probabilistic choice that leads to taking a different path in the model once the activity completes. The place called is initialized to cnt, the number of total HTTP requests for each test case. The rate of the Tarrival activity gives the rate of arrivals per unit of time, which is equal to tps. The HTTP thread instance pool in JBoss is represented by the Threadpool place, which is initialized to maxthread extracted from the configuration file named server.xml. Once Tarrival completes, the token generated, through the output gate OG, shown as a solid triangle, will be deposited in either Start or Blockedrequest. OG represents the conditions for rejecting HTTP requests. For instance, if Start has reached its maximum capacity, OG will redirect the token to Blockedrequest; otherwise the token is added to Start. The rejected requests are accumulated in Down via T3 activity. Activity TJW, which represents the service rate of the application server, is enabled only if Threadpool and Start are not empty. When TJW is activated, a token in Threadpool representing an available thread in JBoss is allocated to a request in Start. Once the request is serviced, the thread, through T1, is released to Threadpool to be used by a next request. In case the server fails to service the request (lower case of TJW), the allocated thread is returned to Threadpool via T2. The failure probability (lower case) of T1 is the probability that the service request is failed by means other than failures caused by the application server, such as a failure in the network or unregistered service. These failure T1 T2 T3 Down probabilities are set to zero but can be set to non-zero values. Table I shows the parameters and their values used in the SAN model. The impulse reward functions count the number of activities that complete. The number of activations at Tarrival shows the total number of requests that has entered the Petri net model. Similarly, the number of activation completions at T3 represents the number of requests rejected. Let be the number of these rejections, i.e. the number of T3 activations. TABLE I. Parameter name s Tarrival rate Threadpool OG TJW rate T1 probability case1 1 T1 probability case2 TJW probability case1 1 TJW probability case2 Impulse Reward Functions SAN MODEL PARAMETERS Parameter value Initialized to Cnt (from SoapUI) Initialized to Tps (from SoapUI) Initialized to maxthread (from server.xml in JBoss AS) // threadpool = maxthread // queue = acceptcount from server.xml in // JBoss AS // maxthread default: 25 // acceptcount default: 1 If (Start Mark() < (threadpool + queue)) Start Mark() = Start Mark()+1; Else Blockedrequest Mark() = Blockedrequest Mark()+1; // avg from SoapUI // threshold = 315 If (Start Mark() < Threadpool Mark()) Return ((1/avg) * (Start Mark()); Else Return ((1/avg) * threshold); Total number of requests: If Tarrival fires then return 1; Number of requests rejected: If T3 fires then return 1; Since the threads in JBoss are executed in parallel, the if-clause of the TJW rate in the table ensures that there are enough threads to be allocated to the requests. The elseclause, however, is not intuitive enough. The else-clause should be looked at with the OG condition in mind. JBoss allows a maximum of queue requests to be queued. In other words, if Threadpool is empty, there can be at most queue tokens in Start. Consequently, if there are x < threadpool tokens in Threadpool, the maximum number of tokens in Start is (queue + x). Since the value of x changes depending on the available threads, the maximum tokens in Start, i.e. (queue + x), continuously changes as well. This causes threshold that represents the speed at which the SAN model services the requests to be dynamic. In other words, threshold needs to be throttled each time x changes. This makes it difficult to predict an appropriate threshold value that meets the rejection rate observed by the experiments performed using SoapUI. Thus, the maximum value of Start is set at the fixed value (threadpool + queue). This in turn makes threshold to be a fixed value. As it will be shown shortly, this approach

5 Rate of web services not serviced Throughput Rejection rate has shown that performance of the SAN model is very close to that of the rejection rate reported by SoapUI. III. RESULTS Sixteen tests have been conducted on the banking web service. Each test is run five times and the average of extracted data is considered as experimental data. Table II shows a sample data extracted from SoapUI. For each of the 16 different loads, avg and tps values returned by SoapUI are used in the SAN model (see Table I) and (5). Table III shows the results for the 16 tests obtained from SoapUI, the theoretical equation (5), and from running the SAN model. As it is shown in the table, the worst rejection rate happens when there are 38 users, which is.13.8 =.5, and.14.8 =.6 for the theoretical and the SAN model, respectively. Users (threads ) TABLE II TABLE III. Number of simultaneous users Runs per threa d SAMPLE DATA EXTRACTED FROM SOAPUI Avg Cnt (sec) Numbe r of request s rejected Total time Tps (sec) rejectio n rate REQUEST REJECTION RATE FOR ALL SIXTEEN TESTS rejection rate (SoapUI) rejection rate (rejection est /cnt) rejection rate (rejection SRN /cnt) Fig. 5 displays graphically the request rejection rate for the three different models shown in Table III. The figure shows that the rejection rate of the analytical and the SAN models closely match the ones provided by SoapUI. Another interesting result from the experimental analysis is the throughput. Throughput is computed by: (6) Fig. 6 shows the throughput as a function of users. As expected, the throughput decreases as the users are increased. The tps range reported by SoapUI is from 21 to 45 for an average of about 33, with a standard deviation of about 1. The average throughput is about 22.5, which means on average, 3% of the requests are rejected Figure Figure Number of virtual users Figure 5. HTTP rejection rate. Number of simultanous users Experimental model Analytical model Throughput based on virtual users (extracted from soapui). Failure rate Total unsuccessful rate (HTTP rejections + failures) HTTP rejection rate Web service failure probability Web services rate not serviced using SAN model. The close approximate behavior of the SAN model, as shown in Fig. 5, allows for cases that are more difficult to replicate in real world, e.g. inducing failures in the application server (TJW activity in Fig. 4). As an example, consider Fig. 7 that shows the rate of failures due to HTTP rejections and web service failures. This

6 figure assumes the experiment with 354 simultaneous users with the same arrival rate and average response time shown in Table II. Rrunning the SAN model with different web service failure probabilities produces the rejection rate of about 3%. But the rate of successful web services is greatly impacted if the web service failure probability is greater than 4%. These probabilities are set by changing case 2 of TJW activity (see Table 1). Although TJW is the activity rate of the web services deployed, we are assuming the application server might fail to service some accepted requests but those web services do not cause the server to fail. Otherwise a failure in the server would require the removal of all requests from Start and restart of the server. IV. FUTURE WORK This study has concentrated on estimation of HTTP rejections. The research treated the application server software as a black box and showed that the SAN simulation model closely matches the experiment results. The current research is investigating a white-box approach (Fig. 8) in that a hierarchical SAN model is being created with submodels for the web server (Tomcat), the application server (JBoss), the database, and the web service itself. Since the correctness of the black-box approach has been verified through real test cases, the black-box results can be used to validate the hierarchical approach in that the white-box strategy ought to produce performance similar to those shown in this research. The white-box approach allows for finer granularity of analysis such as the impact of sensitive components on the overall reliability of the system that is otherwise not possible in the black-box approach. SOAP/HTTP SOAP/HTTP Response Web Server Http thread pool Application Server Figure 8. Architecture of JBoss, showing the interaction between layers and shared resource. V. CONCLUSION This study has emphasized on the combination of load testing and simulation modeling to predict rejections of HTTP requests in a banking web service deployed in JBoss. The analytical and the SAN models developed utilize the parameters extracted from the load tests, such as average response time and arrival rate in order to predict the rejection rate of HTTP requests. The accuracy DB DB connection pool Resource release EJB pool Web service Configuration parameters Resource allocation of the models is validated by comparing the results of the models against those of the experimental ones. ACKNOWLEDGMENT This research is funded in part by Department of Defense (DoD)/Air Force Office of Scientific Research (AFOSR), NSF Award Number FA , under the title High Assurance Software. REFERENCES [1] M.N. Huhns, Service-oriented computing: Key concepts and principles, IEEE Internet Computing, pp , Jan-Feb 25. [2] World Wide Web Consortium, [3] S. S. Gokhale, P. J. Vandal, J. Lu, Performance and reliability analysis of web server software architectures, 12 th Pacific Rim Int l Symp on Dependable Computing, 26. [4] R. C. Cheung, A user-oriented software reliability model, IEEE Trans on Soft Eng, vol. 6, no. 2, pp , 198. [5] K. Goˇseva Popstojanova, K. Trivedi, Architecture based approach to reliability assessment of software systems, Performance Evaluation, vol. 45, pp , 21. [6] M. Rahmani, A. Azadmanesh, H. Siy, Architecture-based reliability analysis of web services in multilayer environment, PESOS workshop, 33 rd Int l Conf on Soft Eng, 211. [7] B. Zhou, K. Yin, S. Zhang, H. Jian, A.J. Kavs, A tree-based reliability model for composite web service with common-cause failures, R.S. Chang et. al (Eds.), LNCS 614, pp , 21. [8] D. Zhong and Zhichang Qi, A Petri net based approach for reliability prediction of web services, LNCS 4277, pp , 26. [9] F. Souza, R. Arteiror, N. Rosa, P. Maciel, Using stochastic Petri nets for performance modelling of application servers, Proc of 5th Int l Work. on Performance Modeling, Evaluation and Optimization of Parallel and Distributed Systems, IEEE IPDPS, pp. 1-8, 26. [1] H. Singh, V. Cortellessa, B. Cukic, E. Guntel, and V. Bharadwaj, A bayesian approach to reliability prediction and assessment of component based systems, Proc. 12th Int l Symp on Software Reliability Engineering, pp , 21. [11] A. Filieri, C. Ghezzi, G. Tamburrelli, Run-time efficient probabilistic model checking, Int l Conf Soft Eng, pp , 211. [12] X. Xiao, T. Dohi, Estimating the error rate in an Apache web server system, Int l J of Soft Eng and Its Applications, vol. 4, no. 3, 21. [13] R. D. van der Mei, R. Hariharan, P. K. Reeser, Web server performance modeling, Telecommunication Systems, vol. 16, no. 3/4, pp , 21. [14] J. Cao, M. Andersson, C. Nyberg, M. Kihl, Web server performance modeling using an M/G/1/K*PS queue, Proc of 1th Int l Conf on Telecommunications, pp , 23. [15] Z. Zheng, M. R. Lyu, Collaborative reliability prediction of service-oriented systems, Proc of 32rd Int l Conf in Soft Eng, 21. [16] K. Goseva-Popstojanova, M. Hamill, and R. Perugupalli, Large empirical case study of architecture-based software reliability, Proc Int l Symp Software Reliability Eng, pp , 25. [17] JBoss Application Server, [18] The Duke s Bank Application, javaee/1.4/tutorial/doc/ebank.html. [19] Wireshark, [2] SoapUI, [21] W.H. Sanders, J.F. Meyer, Stochastic activity network: Formal definition and concept, LNCS, vol. 29, pp , 21. [22] Mobius,