8th WSEAS International Conference on APPLIED INFORMATICS AND MUNICATIONS (AIC 8) Rhodes, Greece, August 2-22, 28 Experimental Evaluation of Horizontal and Vertical Scalability of Cluster-Based Application Servers for Transactional Workloads Daniel.F. GARCIA, Rodrigo GARCIA, Joaquín ENTRIALGO, Javier GARCIA, Manuel GARCIA Department of Informatics University of Oviedo Campus de Viesques, 3324 Gijón SPAIN dfgarcia@uniovi.es http://www.atc.univovi.es Abstract: The scalability of the application servers is an essential issue to guarantee the quality of the services provided by any company that sell its products and services through Internet in business-to-business (B2B) environments. This paper deals with the selection of a proper transactional load to evaluate the scalability, in order to obtain representative results that can provide useful insights for a large range of transactional applications. This evaluation work compares the scalability and other related performance metrics when an application server cluster is scaled horizontally, adding new servers, and when it is scaled vertically, adding cores into the servers. Multiple issues related with the proper experimental design to carry out an evaluation work of these characteristics are also presented in this paper. Finally, it is important to remark that there are few works in the literature concerning the scalability evaluation of transactional systems, in spite of the critical importance of the scalability in this kind of systems. Key-Words: Cluster scalability, Horizontal scalability, Vertical scalability, Transactional benchmarks, TPC-App benchmark, Performance evaluation. 1 Introduction Today, more and more companies announce and provide services through Internet. Generally, the servers of the provider company expose web services that can be invoked by the servers of the consumer companies. The interaction between those servers follows the Business-to-Business (B2B) model. The scalability is an essential issue to guarantee the quality of the services provided by a server operating in B2B environments. These transactional servers always have a typical architecture of at least two layers. A front-end layer constituted by an application server that exposes the web services and executes the business logic, and a back-end layer constituted by a database server. These two layers must scale when the workload supported by the server grows. However, the duplication of the resources (cores and/or computers) of the server rarely becomes in the duplication of the transactional processing capacity of the server. To scale the application server, the business software must be replicable in multiple computers and the web service requests have to be balanced between the computers. The database layer is more difficult to scale. Today, there is only one commercial solution to provide a scalable solution in the database tier: the RAC Oracle database. This evaluation work focuses in the scalability of the application server. The scalability depends on the capability of the business logic software to profit from the addition of computational resources. However, the utilization of a business software that is representative of real applications is essential in order to the results of the evaluation are also representative for the current application servers. The scalability of the application server is also limited by the processing capacity of the database server, the communication architecture within the complete server, etc. 2 Related Work Although the server scalability is an essential issue to guarantee the quality of the services (QoS) provided to a growing number of clients, there is a limited research in this area until now. Within this research area, we could consider three lines in which include the research works. One research line is devoted to present general analyses about the scalability. Hill [1] analyses the scalability property relating it with other properties of computer systems like the speedup and efficiency. Williams and Smith [2] compiled the basic theoretical laws to characterize the web application scalability and they presented a very didactical example using a two-layer cluster. Other authors also considered specific scalability metrics, like the isoefficiency [3] and the isospeed [4]. ISSN: 179-519 29 ISBN: 978-96-6766-94-7
8th WSEAS International Conference on APPLIED INFORMATICS AND MUNICATIONS (AIC 8) Rhodes, Greece, August 2-22, 28 Other line has been focused in the development of scalable solutions to build web servers based on clusters. Pai et al. [5] developed a seminal work designing the Locality-Aware Request Distribution (LARD) technique to distribute the requests arriving to a cluster between its specific-purpose nodes. Cherkasova et al. [6] developed the load balancing strategy called FLEX, in which, the web content is partitioned with different granularity levels and it is loaded into the multiple nodes of a cluster. Casalicchio and Colajanni [7] proposed the Multi- Class Round-Robin technique to build a web switch used as the kernel of scalable web servers. Finally, there are a few research works specifically oriented to the evaluation of the scalability of servers. Haddad and Butler [8] carried out experimental studies of scalability in clustered web systems. Beltran et al. [9] evaluated the scalability of eventdriven web servers, but these servers are rarely used in commercial applications, which are commonly based on thread pools. 3 The design of the evaluation methodology In order to perform a correct evaluation of the scalability a proper evaluation methodology must be designed. We follow the general evaluation methodology proposed by Jain [1]. Firstly, a proper scalability metric must be selected, and secondly, the factors that influence in the scalability must be established. The scalability is a property of a system that indicates its ability to process growing workloads by adding new computational resources, typically, adding hardware. In general, a computer system whose performance improves after adding hardware proportionally to the capacity added is said to be a scalable system. In general, the scalability of a system is a property difficult to quantify. Commonly, it is quantified as the capability of a system to increase its throughput under an increased load when the new resources are added. In this work, the scalability will be calculated from increments of throughput. But we will discuss later the particular measurements of throughput that must be used and the operational conditions of the system required to obtain these measurements. There are many factors that affect the scalability of a cluster that is processing a transactional workload. There are system factors, typically classified in hardware and software factors. The system hardware factors are constituted by the architecture and the capacity of the nodes of the cluster, and of curse, the number and the computing power of the elemental devices, like processors, disks or communication channels. The system software factors are constituted by the operating system, the web server used and its web services processing engine, the distributed transaction coordinator, the message services, etc. There are load factors, typically defined by the server application executed by the cluster. This application must be scalable, that is, it must take advantage of any new resources added to the cluster. In this evaluation work, we are primarily interested in evaluating how the addition of hardware resources affects to the scalability. There are to basic manners to scale a computer system: Scale vertically (or scale up) that consists of adding resources to the nodes of the cluster, typically involving the addition of processors (or cores), memory or disks. Scale horizontally (or scale out) that consists of adding more nodes to the cluster. In particular, this evaluation will consider the effect of adding more cores and more nodes to the application server cluster in a unified manner. All the other factors will be maintained at predefined values, selected of such form that the evaluated system can be very representative of the systems used in real B2B environments. Now, we explain the selection of all components of the system used for the scalability evaluation and the rationality of the selections. As server application we have selected the TPC-App benchmark, because today, it is the most representative synthetic application of the real e-business applications used in the current B2B environments. As computing platform, we have selected a cluster of one-to-four nodes for the application server and one node for the database server. All the nodes are modern HP Proliant computers which are representative of the current servers used in e-business systems for medium-large enterprises. For the system software, we have selected the Microsoft Platform for all components: windows server 23 OS, the IIS web server with the processing engine and the SQL Server 2 for the database. 4 The workload: TPC-App benchmark The TPC-App benchmark emulates the activities of a B2B transactional application server with the objective of obtaining a performance index of the server. The benchmark has three main elements: the system under test (SUT), the Remote Business Emulator (RBE) and external services emulator. The SUT exposes 7 services to the RBE: New Customer, ISSN: 179-519 3 ISBN: 978-96-6766-94-7
8th WSEAS International Conference on APPLIED INFORMATICS AND MUNICATIONS (AIC 8) Rhodes, Greece, August 2-22, 28 Change Payment Method, Create Order, New Products, Order Status, Product Detail and Change Item. The most important is Create Order that uses other two processes to emulate the shipping management of items to clients and the stock management of items. The RBE is composed by several multithread processes. Each thread emulates the activity of an EB that request between 1 and 12 services within a business session. When a business session ends, the EB stars a new business session immediately. In order to provide the requested services, the application server (SUT) requests other 4 web services from the external services emulator. More detailed information about the TPC-App Benchmark can be found in [11] and the full specification can be retrieved in the web site of the Transaction Processing Performance Council [12]. 5 The computing platform The platform used to perform the scalability evaluation is shown in figure 1. It shows the application server implemented as a Network Load Balancing (NLB) Cluster. Using the NLB technology, all web service requests arriving to the virtual IP of the cluster are automatically routed to the real IPs of the nodes using a round-robin policy. The routing is totally done by the network interfaces with a minimal overloading of the nodes of the application server. Al the nodes of the application server are HP ProLiant DL 145 G2 computers. They have two AMD Opteron dual core processors running at 1.8 GHz. The height of these computers is of 1U, and therefore, they only can host two SCSI disks. Preliminary experiments have demonstrated that the best load balance between disks is obtained dedicating a disk to the OS, the DTC and the images of the benchmark, and the other disk to the message delivery service based on MSMQ. The database server, the external vendor server and the load injector are HP ProLiant DL 385 G1, with two AMD Opteron dual core processors running at 1.8 GHz. The database server is equipped with 6 SCSI disks and preliminary experiments have demonstrated that the best load balance between disks is obtained dedicating 1 to OS, 1 to DTC, 1 to DB and 1 to DB Log. All the computers have 2 GB of memory and runs the Windows Server 23 OS. All the internal and external communications are supported by Gigabyte Switches. INJECTOR 192.168.1.181 CLIENT CLIENT CLIENT CLIENT CLIENT NETWORK LOAD BALANCING CLUSTER Business Sessions 192.168.1.186 (Cluster Virtual IP) Business Sessions 192.168.1.172 192.168.1.182 192.168.1.183 192.168.1.184 192.168.1.185 EXTERNAL VENDORS SERVER Gigabit Ethernet 192.168.1.18 DATABASE SERVER Fig. 1. Architecture of the computing platform ISSN: 179-519 31 ISBN: 978-96-6766-94-7
8th WSEAS International Conference on APPLIED INFORMATICS AND MUNICATIONS (AIC 8) Rhodes, Greece, August 2-22, 28 6 The Experimental Design To evaluate the scalability we will start with the minimal cluster, 2 servers and 1 core by server. To check the horizontal scalability, we will duplicate the number of servers. To check the vertical scalability we will duplicate and later quadruplicate the number of cores. With this simple experimental design, 6 long experiments must be done. In each experiment, 1 operational points of the application server are measured. In each point, a fixed load (number of EBs) is supported by the server during one hour. As a preliminary step to calculate the scalability, we are interested in the characterization of the maximum throughput that can be obtained in each experiment (with each configuration of the application server) using a single number. A systematic manner to select a value of the throughput representative of the saturation regimen is explained now. In all graphics of figure 2 there are two lines starting from the origin of the axes. The upper line represents the maximum theoretical throughput achievable by the application server as a function of the emulated business supported. The lower line represents a curve with just the half of the maximum theoretical throughput. Both lines are defined by the TPC-App benchmark to delimit the range of the acceptable values of the throughput of the application server. Values of the throughput at the right of the lower line are clearly considered as belonging to the saturation regimen of the server. We will use the measured throughput value nearest to this lower line as a single representative value of the maximum throughput of the server. Because the main objective of this work is to evaluate the influence of the number of servers and the number of cores on the scalability of an application server we have to establish benchmarking conditions which minimize the influence of other factors as much as possible. Therefore, multiple experiments were carried out to find the optimal configuration of system under test. Due to the high processing speed of the current CPUs, the disks easily becomes into the bottleneck of servers. Into the computers of the application server only two disks can be installed. The better disk load balancing is achieved using one disk to support: the operating system, the Distributed Transaction Coordinator (DTC) and the set of images used by the TPC-App benchmark, and the other disk to support the Microsoft Message Queues (MSMQ). The computer of the database server can host until six disks. After analyzing multiple combinations we have used four disks. One devoted to support the operating system and other to support the database files. These two disks show a very low utilization level during all the experiments. A third disk is used to store the database log and a fourth disk is used to support the DTC. These two disks show a similar high utilization values. The network that connect the load injector with the application server reach a 1% of utilization with the most powerful configurations of the application server. Today, using a 1 Mbps switch there is no risk of saturation in the communications. 7 The experimental results This section present the experimental results obtained with this evaluation work. Figure 2 shows the throughput obtained from the cluster for different configurations of the application server. When we select the throughputs using the procedure described previously we obtain: Nodes Cores / Node 2 4 1 123,55 (15 EB) 238,4 (28 EB) 2 236,11 (28 EB) 278,83 (32 EB) 4 284,8 (32 EB) 284,8 (32 EB) If we divide all the throughputs by the throughput measured for the less powerful configuration we obtain the following scalability metrics: Nodes Cores / Node 2 4 1 1, (1) 1,93 (2) 2 1,91 (2) 2,26 (4) 4 2,31 (4) 2,31 (8) The numbers between parentheses indicate the perfect or linear scalability that we should obtain under ideal conditions. When the graphics of the figure 2 are analyzed by columns, we evaluate the vertical scalability and when they are analyzed by rows, we evaluate the horizontal scalability. The vertical scalability of the application server can be defined as the increment of throughput obtained when the resources (cores) of the nodes of the application server are incremented. The horizontal scalability of the application server can be defined as the increment of throughput obtained when the number of nodes of the server is incremented. The metrics obtained indicate that we will obtain practically the same scalability if the server is scaled horizontally or vertically. The 9 percentile of the response times are shown in the figure 3. They grow progressively with the increment of the server load because the CPUs of the servers reach the 1% of the utilization rapidly and the data server does not become the bottleneck resource. This situation is shown in the graphics 2s-1c/s, 2s-2c/s and 4s-1c/s of the figure 3. ISSN: 179-519 32 ISBN: 978-96-6766-94-7
8th WSEAS International Conference on APPLIED INFORMATICS AND MUNICATIONS (AIC 8) Rhodes, Greece, August 2-22, 28 4 35 3 25 2 15 1 5 4 35 3 25 2 15 1 5 4 35 3 25 2 15 1 5 2 Servers - 1 Core by Server 1 2 3 EB 4 2 Servers - 2 Cores by Server 1 2 3 EB 4 2 Servers - 4 Cores by Server 1 2 3 EB 4 4 35 3 25 2 15 1 5 4 35 3 25 2 15 1 5 4 35 3 25 2 15 1 5 4 Servers - 1 Core by Server 1 2 3 EB 4 4 Servers - 2 Cores by Server 1 2 3 EB 4 4 Servers - 4 Cores by Server 1 2 3 EB 4 Fig. 2. Throughput results of the experiments However, only the response time of the Create Order web service is noticeably affected by the increment of the server load when the data server gradually becomes the bottleneck resource. This situation is shown in the graphics 2s-4c/s, 4s-2c/s and 4s-4c/s of the figure 3. 8 Conclusion This paper has shown that the scalability of small application servers is quasi-linear when they process transactional workloads, but only when the CPUs of the application server cluster are the bottleneck resources. When the data server becomes the bottleneck resource, it limits the scalability of the application server drastically. The visual comparison of the throughputs and the response times indicate that the horizontal and vertical scalabilities are practically the same for small cluster-based application servers. This paper has also established the principles to configure these common transactional systems properly in order to obtain the correct scalability metrics. The key aspect is to avoid the other resources than CPUs became bottlenecks during the evaluation experiments. References: [1] M.D. Hill, What is Scalability?, ACM SIGARCH Computer Architecture News, Vol.118, No.4, 199, pp.18-21. [2] L.G. Williams, C. Smith, Web Application Scalability: A Model-Based Approach, Computer Measurement Group Conference (CMG 4), 24. [3] L. Pastor, J.L. Bosque, An Efficiency and Scalability Model for Heterogeneous Clusters, IEEE International Conference on Cluster Computing (CLUSTER 1), 21, pp.427-434. ISSN: 179-519 33 ISBN: 978-96-6766-94-7
8th WSEAS International Conference on APPLIED INFORMATICS AND MUNICATIONS (AIC 8) Rhodes, Greece, August 2-22, 28 9- Percentile SIRT (msec) 6 5 4 3 2 1 2 Servers - 1 Core by Server 9-Percentile SIRT (msec) 6 5 4 3 2 1 4 Servers - 1 Core by Server 9-Percentile SIRT (msec) 6 5 4 3 2 1 1 2 3 EB 4 2 Server - 2 Cores by Server 1 2 3 EB 4 9-Percentile SIRT (msec) 6 5 4 3 2 1 1 2 3 EB 4 4 Servers - 2 Cores by Server 1 2 3 4 EB 9-Percentile SIRT (msec) 6 5 4 3 2 1 2 Servers - 4 Cores by Server 9-Percentile SIRT (msec) 6 5 4 3 2 1 4 Servers - 4 Cores by Server 1 2 3 EB 4 1 2 3 4 EB Fig. 3. 9 Percentile response time of the seven web services of the TPC-App benchmark [4] Y. Chen, X.-H. Sun, STAS: A Scalability Testing and Analysis System, IEEE International Conference on Cluster Computing (CLUSTER 6), 26, pp.388-397. [5] V.S. Pai, M. Aron, G. Banga, M. Svendsen, P. Druschel, W. Zwaenepoel, E. Nahum, Locality- Aware Request Distribution in Cluster-Based Network Servers, 8 th International Conference on Architectural Support for Programming Languages and Operating Systems (LOS 98), 1998, pp.25-216. [6] L. Cherkasova, FLEX: Load Balancing and Management Strategy for Scalable Web Hosting Service, 5 th IEEE Symposium on Computers and Communications (ISCC ), 2, pp.8-13. [7] E. Casalicchio M. Colajanni, Scalable Web Clusters with Static and Dynamic Contents, 2 nd IEEE International Conference on Cluster Computing (CLUSTER ), 2, pp.17-177. [8] I. Haddad, G. Butler, Experimental Studies of Scalability in Clustered Web Systems, 18 th International Parallel and Distributed Processing Symposium (IPDPS 4), 24, pp.185-193. [9] V. Beltran, D. Carrera, J. Torres, E. Ayguade, Evaluating the Scalability of Java Event-Driven Web Servers, IEEE International Conference on Parallel Processing (ICPP 4), 24, pp.134-142. [1] R.K. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation and Modeling, John Wiley & Sons, 1991. [11] D.F. Garcia, J. Garcia, M. Garcia, I. Peteira, R. Garcia, P. Valledor, Benchmarking of Web Services Platforms: An Evaluation with the TPC- App Benchmark, 2 nd International Conference on Web Information Systems and Technologies, 26, pp.75-8. [12] Transaction Processing Performance Council, http://www.tpc.org ISSN: 179-519 34 ISBN: 978-96-6766-94-7