}w!"#$%&'()+,-./012345<ya

Transcription

1 }w!"#$%&'()+,-./012345<ya Masarykova univerzita Fakulta informatiky Performance comparison of JBoss integration platform implementations Master Thesis Elena Medvedeva Brno, May 2014

2 Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Elena Medvedeva Advisor: Mgr. Marek Grác, Ph.D. ii

3 Acknowledgement I would like to thank my technical supervisor from Red Hat, Ing. Pavel Macík for sharing his vast experience in the field of performance testing, and for valuable pieces of advice given to me during the preparation of this work. I am grateful to my supervisor Mgr. Marek Grác, Ph.D. for the comments and consultations. Besides, I want to thank all my colleagues who work with me in the JBoss Fuse QA team, for support, and the company Red Hat, which provided me the opportunity to write this thesis. iii

4 Abstract We present the results of our investigation in the field of performance testing of JBoss integration platforms to compare performance of JBoss Fuse and SwitchYard. We design a basic set of performance scenarios to cover basic usage patterns of integration platforms. We formalize the scenarios that are already implemented for SwitchYard, and we develop three new scenarios. All fifteen scenarios are implemented for JBoss Fuse using PerfCake and tweaked to be optimal from the performance point of view. In addition performance test execution is automated in distributed environment using Jenkins and SmartFrog. Finally, we collect the results of performance testing, and compare the performances of two JBoss integration platform implementations JBoss Fuse and SwitchYard. iv

5 Keywords performance testing, integration platforms, system integration, JBoss Fuse, JBoss SwitchYard, PerfCake, SmartFrog, Apache ActiveMQ, Apache Camel, Apache CXF, web services, Camel routes v

6 Contents 1 Introduction Enterprise service bus (ESB) Evolution of Java Applications Service-Oriented Architecture concept Definition of Enterprise Service Bus (ESB) Integration framework ESB Integration Suite JBoss integration platforms JBoss Fuse Apache Camel Apache ActiveMQ Apache CXF Apache Karaf Fuse Fabric SwitchYard Performance testing Types of performance testing Load testing Stress testing Soak or stability testing Configuration testing Smoke testing Performance metrics Performance standards Tasks to fulfill during performance testing Open source performance testing tools Testing environment and test automation Testing environment Test automation Test automation tools Task formulation and test automation implementation Implementation of test automation Environment characteristics Performance tests scenarios design and implementation 31 1

7 7.1 General architecture of tests Performance test scenarios design Scenario: HTTP exposed custom service Scenario: SOAP exposed custom service Scenario: JMS exposed custom service Scenario: HTTP exposed content based routing using XPath Scenario: HTTP exposed content based routing using RegEx Scenario: HTTP exposed content based routing using Rules Scenario: HTTP exposed Services implementing Scatter-Gather pattern Scenario: HTTP exposed Services implementing Splitter-Aggregator pattern Scenario: Service orchestration Scenario: SOAP exposed XML message transformation using XSLT Scenario: SOAP implementation of a web service using JAX-WS Scenario: SOAP implementation of a web service using JAX-WS secured by WS-Security Scenario: SOAP web service proxy Scenario: Method GET of a RESTful web service implementation Scenario: Method POST of a RESTful web service implementation Implementation of performance scenarios for JBoss Fuse HTTP exposed custom service SOAP exposed custom service JMS exposed custom service HTTP exposed content based routing using XPath HTTP exposed content based routing using RegEx HTTP exposed content based routing using Rules HTTP exposed Services implementing Scatter- Gather pattern HTTP exposed Services implementing Splitter- Aggregator pattern

8 7.3.9 Service orchestration SOAP exposed XML message transformation using XSLT SOAP implementation of a web service using JAX- WS SOAP implementation of a web service using JAX- WS secured by WS-Security SOAP web service proxy Methods GET of a RESTful web service implementation Methods POST of a RESTful web service implementation Results Concluding the results Effort allocation and issues encountered Conclusion

9 1 Introduction Nowadays, the market demands software applications to become more and more complex, to process data from various providers, presented in different formats. Integration platforms were introduced to help developers create complex data interaction and processing in their applications, using standardized approach. Integration platforms are used to integrate different applications and services. In this work we will consider two JBoss projects implementations of integration platforms: JBoss Fuse and SwitchYard. These projects provide similar functionality. So it is useful to find out which implementation works better from the performance perspective, and under which circumstances. The goal of this work is to compare the performance of those two integration platform implementations. The thesis consists of eight chapters. The Chapter 1 briefly introduces the topic of the thesis. The Chapter 2 provides an overview of the notions service oriented architecture and enterprise service bus. In the Chapter 3 the JBoss integration platforms JBoss Fuse and Switch- Yard are described in details, focusing on JBoss Fuse in particular. The technologies for working with JBoss Fuse are introduced such as Apache Camel, Apache ActiveMQ, Apache Karaf, Apache CXF. Methodology of performance testing is defined in the Chapter 4. The Chapter 5 covers the theory underlying test automation and the design of test environment. In the Chapter 6 we describe the process of test automation implementation for JBoss Fuse performance tests. We also define the environment characteristics where the tests were executed. The Chapter 7 focuses on formulation of test scenarios for performance comparison of the platforms. Besides, description of implementation of scenarios for JBoss Fuse is in this chapter. Chapter 8 contains the results of the performance measurements, the analysis of the results, and description of the allocation of the effort, while working on the master thesis. This chapter also describes the issues faced during the work. 4

10 2 Enterprise service bus (ESB) 2.1 Evolution of Java Applications According to the book [5] the first Web sites were build mainly from static content. Static content is delivered to the user exactly as stored, unlike dynamic content which is generated by web server at the time the user requests the page. But when an owner of the website wanted to modify the page, he had to modify the physical HTML file. Each operation required too much effort, and in order to solve this issue, tools and frameworks for dynamic Web content generation appeared. In 1997 servlet specification was released by Sun Microsystems. A servlet is a Java program that runs within a Web server. Servlets receive requests and respond to requests from Web clients, usually across the HyperText Transfer Protocol (HTTP) 1. Servlets were used to generate dynamic content (pages). But soon developers understood that it was not comfortable to put presentation details into java code. Due to this fact, in 1999 Sun released JavaServer Pages (JSP) specification. JSP technology enables to integrate Java code into HTML tags to generate pages dynamically. JSP technology makes available all the dynamic capabilities of Java Servlet technology but provides a more natural approach to creating static content[10]. But it was too complicated to put all business logic into JSP. As a result of all above, the design pattern Model-View-Controller (MVC) was created. In the variation of MVC JSP were used for presentation(view), servlets represented business logic (Controller). As there was separation in layers, the applications became more maintainable and flexible. Apart from that, Enterprise JavaBeans (EJB) were introduced to deal with persistence, transaction integrity, and security in a standard way [10]. As Java gave developers a lot of flexibility while creating application, design patterns where introduced to represent best practices in application development

11 2. Enterprise service bus (ESB) 2.2 Service-Oriented Architecture concept As the time went by, and business application became more and more complex, new architectural concept for development of Java applications emerged - Service-Oriented Architecture (SOA). In simple words SOA, and Web services facilitated the interoperability between frameworks or applications written in different languages and running on different operating systems. Interoperability is the ability of making systems and organizations to work together (inter-operate). The following definition of SOA was produced by the SOA Definition team of The Open Group SOA Working Group. Service-Oriented Architecture (SOA) is an architectural style that supports service-orientation. Service-orientation is a way of thinking in terms of services and service-based development and the outcomes of services. A service is a logical representation of a repeatable business activity that has a specified outcome (e.g., check customer credit, provide weather data, consolidate drilling reports), and Is self-contained; May be composed of other services; Is a black box to consumers of the service[14]. Among the basic principles of SOA are standardization and service loose coupling. Standardization: means that service is defined by one or more servicedescription document, and its implementation can be easily substituted by another implementation, which satisfies the same contract. Service loose coupling: refers to the minimization of dependencies inside the system. Web services is a collection of technologies that implements a serviceoriented architecture. Web services is a platform- and technology-agnostic collection of specifications by which services can be published, discovered and communicate with one another [5]. 6

12 2. Enterprise service bus (ESB) 2.3 Definition of Enterprise Service Bus (ESB) An enterprise service bus (ESB) is a software architecture model used for designing and implementing communication between mutually interacting software applications in a service-oriented architecture (SOA). ESB was introduced as a solution for communication of different applications even within different companies. Figure 2.1: An example of an enterprise service bus. Currently there is no standard definition of the term ESB. In this work we will distinguish ESB from integration framework and integration suite according to the paper Choosing the Right ESB for Your Integration Needs by Kai Wähner[20] Integration framework Integration framework(if) implements Enterprise Integration Patterns (EIP) which are designed to integrate application in a standardized way. Examples of Java-based integration frameworks include Apache Camel and Spring Integration. The usage of IF reduces developer efforts. IF 7

13 2. Enterprise service bus (ESB) supports different protocols, and technologies. It also uses EIP to specify the way of communication between those technologies. Integration framework also simplifies understanding of the code for integration ESB Enterprise Service Bus (ESB) is based on integration framework and adds tools for deployment, administration and monitoring at run-time. Besides, it offers graphical editors for various integration scenarios. Sometimes, graphical editors use drag and drop, when the source code is automatically generated. ESB allows integration at higher abstraction level then integration framework. Examples of ESB are JBoss Fuse ESB, SwitchYard, both developed mainly by JBoss Community, which will be considered later, and also Mule ESB. All these ESB are open source. Among proprietary solutions the most prominent are Oracle Service Bus and IBM WebSphere ESB Integration Suite Integration suite adds to ESB tooling for the following features: Business Process Management (BPM), Business Activity Monitoring (BAM), Master Data Management (MDM), and can add a Repository. Business Process Management refers to a systematic approach for definition, description, improvement of an organization s business processes. It uses specific language for description of business processes. Business Activity Monitoring is an application for providing real time data about the status and results of processes and transactions. It provides flexible configuration of monitoring the work of, for example, running services. Master Data Management provides a single point for managing important data of application. Repository is used for version controlling and dependency monitoring of currently deployed application in the suite. 8

14 3 JBoss integration platforms This thesis is dedicated to the performance comparison of two JBoss projects: JBoss Fuse and SwitchYard. First, I would like to take a closer look at them. 3.1 JBoss Fuse According to the classification from the previous chapter JBoss Fuse is a pure enterprise service bus. It is based on Apache Camel, as an integration framework, Apache CXF as a services framework and Apache ActiveMQ as messaging framework. All those frameworks are running on Apache Karaf, which provides OSGi-based container where applications and components can be deployed. Apart from that JBoss Fuse contains Fuse Fabric for managing multiple containers running on different hosts. Fuse IDE is a plugin for Eclipse 1 which offers possibility for design time editing of Camel routes, and advanced run-time and debugging features. Besides, there is Hawtio - web based management console. Figure 3.1: Technologies comprising JBoss Fuse

15 3. JBoss integration platforms Apache Camel As an integration framework Apache Camel, provides a way for developers to integrate different systems into one application. Different protocols can be used for communication with different systems. Camel allows developers to create routing rules, which determine the source of the messages, what should happen with them during routing (some transformations) and destination (where to send the resulting messages). The messages are used for communication between the systems inside the application. In Camel the routing rules are defined regardless of the protocol and data-type the systems are using, thus creating another level of abstraction. The transformation to the required protocol happens automatically. Currently, there are more than 150 connector implemented in Camel to different systems and technologies, including the most simple as files, URLs, POJO (plain old java objects, an ordinary Java object) to more complex, as web services, SAP, Facebook, Salesforce and ActiveMQ, or other messaging framework. Routing rules can include elements of Enterprise integration patterns (EIPs), which allows developers to create complex routing, suitable for business process management. The routes can be described using different languages, such as Java, or XML (in Blueprint XML or in Spring XML)[7]. Blueprint XML and Spring XML are two dependency injections frameworks supported by JBoss Fuse[17]. The advantage of Blueprint XML is that it automatically resolves the dependencies at run time, if the project was packaged as an OSGi bundle. To sum it up, Apache Camel main goals are: to offer concrete implementations of all the widely used EIPs; to provide connectivity to a great variety of transports and application programming interfaces (APIs)[1]. The advantages of Camel include the large community of users and developers, and its extensible architecture which, allows 3rd party developers to add connectors to new protocols Apache ActiveMQ Apache ActiveMQ is an example of Message Oriented Middleware (MOM), an application which uses messages for communication between its 10

16 3. JBoss integration platforms parts, which provides the advantage of loose coupling. Java Message Service(JMS) is a standard specifying how Java application should send, receive and create messages. Using this standard, a JMS client written in one technology can exchange messages with another JMS client, using JMS provider. JMS provider is an implementation of the JMS interfaces which is ideally written in pure Java.[book ActiveMQ in Action] According to JMS specification, JMS producer is created by client application to create and send JMS messages, and JMS consumer created by a client application to receive and process JMS messages. JMS provides two models of communication: point-to-point; publish/subscribe. In point-to-point communication the JMS producer sends messages to the JMS queue. The messages stay in the JMS queue, until consumer gets them from the queue. When the message is consumed it is removed from the queue. The key point is that the message is delivered to the only one JMS client. JMS queue is the area where messages are stored till they will be consumed by JMS consumer or expire. During publish/subscribe communication the messages is send to the JMS topic, and from the topic it is received by all clients, who where subscribed to the topic. The main difference from point-to-point communication is that the messages can get to many consumers(in this model they are called subscribers) at once. Topics don t store messages unless they are explicitly instructed to do so. JMS offers two types of message delivery mode: persistent and not persistent. Persistent messages are delivered once-and-only-once to destination, even if the provider fails to deliver it first time. This puts more overhead on database of JMS provider, since the message is stored till it is delivered to destinations. Non-persistent messages will be delivered at most once, which means if provider fails to deliver the message, he will not try again. In this case there is no overhead on provider, and it increases performance, but decreases reliability. The control over the messages and routing them to correct queues or topics and other activities on the provider side are done by message broker. A message broker is an architectural pattern for message validation, message transformation and message routing[20]. 11

17 3. JBoss integration platforms ActiveMQ is an open source, JMS 1.1 compliant, message-oriented middleware (MOM) from the Apache Software Foundation that provides high-availability, performance, scalability, reliability and security for enterprise messaging[19]. ActiveMQ provides implementations of JMS client and a message broker Apache CXF In order to make a software system available to another systems over the Web one can create a web service - a software system with standard interface, which is available on some Web address (URL). Apache CXF is one of the leading standard-based web services framework whose goal is to simplify web services development.[book Apache CXF Web service development] Web services can be developed using two main approaches: using the Simple Object Access Protocol (SOAP); using the Representational State Transfer (REST) architectural style. Apache CXF supports both of these approaches. Simple Object Access Protocol (SOAP) is a protocol for exchanging XML-based messages over a network, typically using HTTP protocol. [book Apache CXF Web service development] SOAP messages consists of body and header, where header stores information about security, transactions and other context related information, while the body contains application data ( or payload). Web Services Description Language(WSDL) is an XML-based language for description of web services. Currently there are two standards WSDL 1.1 and WSDL 2.0. WSDL file defines the operations(functions and procedures), which a web service provides, input and output arguments, and their types, and also the exact protocol binding(for example SOAP) and endpoint or port binding, typically represented by a simple HTTP URL string. In SOA an endpoint is an entry point to a service, a process, or a queue or topic destination. So, by World Wide Web Consortium (W3C): A Web service is a software system identified by a URI whose public interfaces and bindings are defined and described using XML (specifically WSDL). Its 12

18 3. JBoss integration platforms definition can be discovered by other software systems. These systems may then interact with the web service in a manner prescribed by its definition, using XML-based messages conveyed by Internet protocols. Java API for XML Web Services (JAX-WS) is a specification designed to simplify the construction of primarily SOAP-based web services and web service clients in Java[3]. This is one of the most important approach to web service development. Representational State Transfer (REST) is a style of building a distributed application architecture, which is often used to build web services. Systems which implement REST are called RESTful systems. REST architecture operates resources. Each resource has an identifier within the system. For example, for the HTTP resource identifications are URL-addresses. Web services developed using the REST approach are viewed as resources and identified by their URI. Method or function of a web service is an action, and actions are identified by for commands: GET, POST, PUT and DELETE. Java API for RESTful Web services (JAX-RS) is a specification that determines the semantics to create web services according to REST architectural style[3]. This technology allows to expose any java classes as web services using annotations and a special servlet that is provided by implementations of JAX-RS. JAX-WS and JAX-RS specifications provide a set of annotations to convert POJOs as SOAP and RESTful web services, what makes creation of web services easy. But unlike JAX-WS in JAX-RS no need for additional xml configurations, as WSDL is required to implement a web service. RESTful web services are considered to be simpler technology for implementation, then SOAP based web services. Besides while following REST architecture style there is less coupling between service and its client, so when something has changed in the web service contract, developers don t necessarily have to change the client s implementation, unlike with SOAP based web services. In addition, RESTful implementations there is possibility to send messages between service and client in other data formats apart from XML, such as Java Script Object Notation (JSON). CXF implements JAX-WS and JAX-RS specifications. Also CXF provides a set of API to expose POJOs as web services and create web service clients. 13

19 3. JBoss integration platforms Apache Karaf Apache Karaf provides a container for managing life cycle of OSGibundles. The OSGi (Open Service Gateway initiative) is a set of specifications that define a dynamic component system for Java, introduced by OSGi Alliance[16]. This specification is used to construct Enterprise applications, and complex desktop applications, such as Eclipse SDK with pluggable architecture. According to this specification, the basic unit is an OSGi-bundle. OSGi bundles contain java-classes and other resources that together can implement some functions as well as provide services and packages for other bundles. Technically an OSGi bundle is a jar archive with a special file, called manifest, which describes the the classes and interfaces to be exported and imported by the bundle, and also includes information about the name and version. Maven allows automatic generation of manifest file using mvn-bundle-plugin. Bundle can be in several states in the OSGi system: Installed: the bundle was successfully installed into the system. Resolved: all dependencies were resolved. For the bundle all Javaclasses and those bundles, on which it depends are available. This status indicates that the bundle is ready to start. Starting: the bundle is starting. Active: the bundle was started successfully. Stopping: the bundle is stopping. Uninstalled: the bundle was stopped/or didn t started. The bundle was deleted, so it can t got to any other state except installed again. There are several frameworks which implement OSGi technology. Among them are Apache Felix framework and Eclipse Equinox OSGi frameworks[2]. Apache Karaf can be configured to use each of them and adds additional functionality. Among the most important features are hot deployment (automatically starts all files which are in the deploy directory), versioning, management console, advanced logging, security, 14

20 3. JBoss integration platforms possibility to manage multiple instances of Apache Karaf through the main instance (root), and other features. By default Apache Karaf runs on Apache Felix framework. There is also possibility to deploy OSGi-bundles, war files, FAB or a feature. OSGi-bundle: OSGi-bundle is a jar file with manifest (described earlier). War: War file is a jar archive of a web application.. Fuse application bundle (FAB): FAB is a jar file, which is converted by Fuse ESB to and OSGi-bundle after installation. Feature: A feature is a way of aggregating multiple OSGi bundles into a single unit of deployment[17]. Feature is represented by an XML file, which contains maven coordinates of the bundles and other features, which are included into the feature Fuse Fabric Fuse Fabric is a technology layer that supports the scalable deployment of Fuse ESB Enterprise containers across a network.[fuse documentation] Fuse ESB Enterprise container is a container for deployment OSGi-bundles,FAB, and war files, running on Fuse kernel. Fuse Fabric provides a possibility to manage and monitor multiple Karaf container instances in the cloud (on multiple hosts). Test scenarios were developed for comparison of SwitchYard and JBoss Fuse. But there is no alternative to Fuse Fabric in SwitchYard. 3.2 SwitchYard SwitchYard is a component-based development framework focused on building structured, maintainable services and applications using the concepts and best practices of SOA[9]. It is also an enterprise service bus (ESB), according to the classification described in the Chapter 2, though there is support for business process management (BPM). SwitchYard is an open source JBoss project. 15

21 3. JBoss integration platforms It uses Apache Camel as integration framework, which was described earlier. HornetQ is used as a messaging framework. It is an example of Message Oriented Middleware, and JMS provider implementation. It performs the same functions in SwitchYard, as ActiveMQ in JBoss Fuse. SwitchYard runs on JBoss Application Server(JBoss AS).The JBoss Application Server is a Java EE application server platform for developing and deploying enterprise Java applications, web applications, and web portals[8]. SwitchYard supports integration with JBPM, which is a business process management framework, used for service orchestration and human task integration expressed as BPMN 2[9]. BPMN 2 (Business Process Model and Notation) is a graphical representation for specifying business processes in a business process model[8]. There is an Eclipse plugin for visual representation of the integration design for SwitchYard. SwitchYard offers a full support for Java EE 6. I will focus mainly on description of JBoss Fuse, as implementation of performance test scenarios for SwitchYard were already created by SwitchYard team. Besides, the theoretical description of the technologies used in the test scenarios was made in the Section 3.1 of this Chapter. 16

22 4 Performance testing Software testing is a process of examining the software that comprises two goals: to show to developers and customers that the program meets the requirements, to identify situations in which the behavior of the program is incorrect, inappropriate or inconsistent with the specification. According to the subject of testing the following categories could be distinguished: functional testing; performance testing; usability testing; security testing; localization testing; compatibility testing; All types of tests defined above are dedicated to ensure that the application possesses characteristics, required by standard ISO 1, which are used for evaluation of software quality. ISO/EIC 9126 standard consists of six main criteria: Functionality: Are the required functions available in the software? Reliability: How reliable is the software? Usability: Is the software easy to use? Efficiency: How efficient is the software? Maintainability: How easy is to modify the software? Portability: How easy is to transfer the software to another environment?

23 4. Performance testing Performance is a very important characteristic of any software application. It provides information about efficiency and reliability of the software. A well-performing application allows users to perform tasks without significant delay and irritation. According to the book Pro Java EE 5 Performance Management and Optimization by S. Haines [5] the impact of poor performance result in the lost productivity, which can also lead to lost of customer confidence and credibility and as a result it can lead to decrease in revenue. When using a poorly performing software as internal application, the company is paying its employees for waiting for software to respond. Moreover, troubleshooting takes more time, and happens more often in the poorly performing applications. The customers of such company can become not confident in the services of the company, if the employees deliver them after the deadline. As a result, they will select another better performing companies. Before proceeding to performance testing it s very important to understand the goals of performance testing in the exact case. The possible goals include: to make sure that all basic transactions of the system meets some predefined performance criteria; to compare several systems to find out which system is better in terms of performance and for which transaction; to find out which parts of the system have the worst performance; 4.1 Types of performance testing There is no single approach to definition of types of performance tests. The following areas are distinguished: load tests, stress tests, endurance or soak or stability tests, configuration tests, smoke tests[12] Load testing Load testing is the classic form of performance testing, where the application is loaded up to the specified level but usually not further. Load testing is usually performed in order to evaluate the behavior of the 18

24 4. Performance testing application on a given expected load. This load can be, for example, the expected number of concurrent users of the application, creating a specified number of transactions per time interval Stress testing Stress testing is used to measure the upper limits, or the sizing of the infrastructure. Thus, a stress test continues until something breaks, e.g. no more users can log in, application becomes unavailable, etc. It is important to know the upper limits of the applications, especially, if future growth of the application traffic is expected Soak or stability testing Stability testing is dedicated to identify problems which occur only after a large period of time.the aim of this kind of testing is to make sure that the application will work as expected under specified load during the long time-frame. A classic example of problem which can be found during such tests is a slowly developing memory leak or a gradual slowdown in the number of transactions which are executed per timeframe. Thus, memory utilization is monitored to detect potential leaks Configuration testing The aim of the configuration tests is to define how different types of system s configuration can influence the productivity of the system. This tests can be also combined with load, stress or stability tests Smoke testing In performance testing this term refers to testing only those transactions that has been affected by a code change. 4.2 Performance metrics The performance of the application can be assessed using the following criteria: 19

25 4. Performance testing Response time specifies the amount of time the user has to wait to get the response from the application. In terms of performance testing, this is the time between the user s requesting response from the application and a complete reply, arriving at the users workstation. Throughput defines the number of things we can accomplish in a time period. In Java EE terms, request throughput is a number of requests a system can service in a second. The goal is to maximize request throughput and to measure it against the number of simultaneous requests. Request throughput reflects the response time[5]. Resources utilization defines the percentage of the capacity of the resource is being used by the application. This results help to analyze the work of application, and help to find the root cause of performance degradation, if any. The most important metrics are CPU usage and memory usage. Among others for Java applications are thread pools, JDBC connection pools, caches, JMS servers and others. In case of Java applications, all Java objects are placed in a part of the memory called the heap. When the heap becomes full with references to objects which are no longer in use, the memory is cleaned by a special automatic process called Garbage Collector. Time spent by the CPU to clean memory can be significant, in case the process has occupied all available memory (in Java - the so-called Full GC ) or when the process allocated a lot of memory, which now needs to be cleaned. While Garbage Collector is cleaning the memory, the access to the allocated pages of memory is blocked, which may affect performance. Besides, as specified in the Oracle documentation, when the heap memory is fragmented (there are a lot of small free spaces in the heap, but allocation of large objects is hard or even impossible), the process of compaction is executed, which moves objects closer together, thus creating larger free areas for new objects. Compaction is performed while all Java threads are paused, what can also influence performance of the application. 20

26 4. Performance testing Availability defines how much time the application is available to the users. The application is not available to the users when they are completely unable to effectively use the application. 4.3 Performance standards According to the article by Jakob Nielsen Response Times: The 3 Important Limits [13], there are 3 main time limits (which are determined by human perceptual abilities) to keep in mind when optimizing web and application performance: 0.1 second: Time limit for a user to have illusion that he is manipulating objects in the user interface. 1 second: Time limit for a user not to get irritated by a delay in the application processing time, though the user will notice the delay. This is the time-frame during which a user can perform operations which require him to remember information throughout several responses. 10 seconds: Limit for users keeping their attention on the task. more than 10 seconds: After waiting more than 10 seconds average user will switch his attention to another task. 4.4 Tasks to fulfill during performance testing According to the book The Art of Performance Testing by I. Molyneaux [12], the following tasks should be undertaken during performance testing: Gather performance requirements from the customer. Develop a high-level plan, including requirements, resources, time-frames and milestones. Decide on the test team involved into performance testing. Design test environment for performance testing. 21

27 4. Performance testing Choose a testing tool Perform the Proof of Concept for the chosen tool. Develop a detailed performance test plan that includes all dependencies and associated time lines, detailed scenarios and test cases, workloads, and environment information. Configure the test environment. Strive to make test environment a close approximation to the live environment. Ideally identical hardware to the production platform, router configuration, quiet network, because the results shouldn t be influenced by other users, deployment of server instrumentation, database should be populated realistically in terms of content and sizing, etc. Transaction scripting. Implement each test scenario in the test plan. Run performance test execution. Run and monitor tests enough times to make sure the results are not effected by some unaccounted factor. Analyze results, report, retest. Document all necessary information about each test run, investigate problems, apply corrective actions, retest if necessary. 4.5 Open source performance testing tools Performance testing of a web server application can be performed using automated tools such as: JMeter: This is the most popular tool for performance testing. It is written in Java, and supports testing of the following technologies: JDBC / FTP / LDAP / SOAP / JMS / POP3 / HTTP / TCP. It allows creation of a large number of requests from different computers and monitoring the process of testing from one of them. JMeter can be used to make a graphical analysis of performance. The tool supports extensions via plugins. 22

28 4. Performance testing PerfCake: This is relatively new framework for performance testing. The first release appeared in It is written in Java and the following technologies can be tested using it: HTTP, REST, JMS, JDBC, SOAP, socket, file. PerfCake uses XML description file for configuration of tests. The tool can measure throughput, response time and memory consumption of the target JVM. Test results are reported to shell console or CSV. The tool can be extended via plugins 2. ApacheBench: This is also a very popular and one of the most simple in use tools for performance testing. All tests settings are described in the command line, and no configuration files is required. It is single-threaded, and can measure performance only of HTTP queries. ApacheBench was originally designed to test the Apache HTTP Server, but it is generic enough to test any web server. The tool supports GET and POST methods. Results can be generated in CSV format. Curl-loader: Curl-loader is a C-written web application testing and load generating tool. The tool can simulate tens of thousand and hundred users/clients each with own IP-address. It supports the following technologies for testing: HTTP, FTP and TLS/SSL protocols. Besides, user authentication, log in can be tested using Curl-loader and it provides range of statistics[15]. All the tools described above support Linux, also PerfCake and JMeter can run on Windows

29 5 Testing environment and test automation 5.1 Testing environment Testing environment is a combination of configured hardware and software on which the testing team is going to perform the testing of the application[11]. When creating the testing environment, the main goal is to simulate as close as possible the usage of this application in production. Typically, for testing of an enterprise service bus the following configuration is used: Server host: This is a host, where the server application is running. In the case of performance testing, enterprise service bus is running on this host with some testing application deployed to it. This application implements some functionality from test scenario. Client host: On this host runs the testing tool, which is generating the load. The testing tool sends requests over the network to the testing application running on server host, and waits for a response. Database host: Generally, the database used by the ESB is located on a separate host in production. Apart from that there might be a special machine dedicated for the database of testing application. Helper host: This is optional host, which can be used for simulation of some third-party dependencies of the scenario implementation. For example,it can be used to run some web services which are called from testing application. The database and helper hosts are optional, and are used when it makes sense. Another reason, why client-server environment is used for performance testing, is that the testing tool, which generates load, consumes resources, and when placed on a separate host, it doesn t 24

30 5. Testing environment and test automation affect the throughput of the testing system. But there is an influence of the network connecting the hosts on throughput. In order to minimize the network influence, the tests should be run in a dedicated highperformance network. 5.2 Test automation Test automation is a use of special software to run and configure the test and test environment, and collect test results. The software which runs the tests should differ from the software which is tested. Automation is very beneficial for performance testing, as it allows to run preparation of the test in similar flow, when actions for preparation are performed in the same sequence, with the same delay for all scenarios. Besides, the tests can be run faster Test automation tools Jenkins is widely used for test automation. Jenkins, originally called Hudson, is an open source Continuous Integration tool written in Java. Continuous Integration, in its simplest form, involves a tool that monitors a version control system for changes. Whenever a change is detected, this tool automatically compiles and tests the application[18]. Jenkins can download the changes of the tests from the repository, there is a possibility to specify many repositories, and then to run the scripts, where further actions on tests automation are specified. In Jenkins there is a possibility to provide some settings, for tests run, including, the names of the host where to run the scripts, and the version of JDK to use, and others. The client-server test environment requires flexible management of testing process on multiple hosts. This can be done by usage of Smart- Frog. SmartFrog is a powerful and flexible Java-based software framework for configuring, deploying and managing distributed software systems[6]. SmartFrog has it s own language to describe the sequence of actions, which should be implemented to run SmartFrog components. Smart- Frog components are Java classes, which implement predefined interfaces, and are considered by SmartFrog a single unit of work. In the main configuration SmartFrog script, one can specify the interaction 25

31 5. Testing environment and test automation of SmartFrog components, for example to run them in parallel or in sequence, specify the host where to run them, and what should happen with the whole system if one component fails. 26

32 6 Task formulation and test automation implementation The main goal of this diploma thesis is to compare performance of JBoss Fuse and SwitchYard. In order to achieve more fair comparison of performance results, performance of JBoss Fuse and SwitchYard was tested with the same testing tool. PerfCake was chosen as a testing tool, because it matches all requirements that were set by design of performance tests. The performance tests execution was automated on Jenkins, using SmartFrog. I will focus on description of automation for JBoss Fuse, because automation for SwitchYard is implemented in a similar manner and uses the same design model. 6.1 Implementation of test automation The code with test scenarios, the messages which are sent for load generations, and applications, which are deployed to JBoss Fuse to test some functionality (let s call them tested service (TS)), are stored in the repository, managed by version control system. All performance test scenarios for JBoss Fuse are accessible through the Jenkins job. Figure 6.1: Tests automation in Jenkins. In order to rebuild the tests only one click on the link in web browser is required. This will execute sequential run of all tests automatically. 27

33 6. Task formulation and test automation implementation Let s consider the actions which are done during the automated execution of one performance test scenario. 1)The Linux script kills all possible residual processes from previous test runs, including the process of JBoss Fuse, exactly Karaf on the server machine, and PerfCake process from the client machine. Besides, new code changes are upload to the SmartFrog components, scenarios, messages and tested services from the repository. The folders with all code, are located in the folders shared between all machines in the network, so that both host and server could access them. This script is run by Jenkins. 2)After the Linux script is finished Jenkins runs SmartFrog on the client machine, and provides it with its configuration script. The further steps are described in the SmartFrog configuration script. 3)SmartFrog script performs the following sequence of actions, as drown on the picture below. Figure 6.2: Components sequence execution in SmartFrog. SmartFrog runs PrepareServer component at the server host, and PrepareTestsAtClient at the client host. The execution of components at host and server start at the same time and in parallel. When PrepareServer component is finished, SmartFrog starts PrepareTestsAt- Server component at server host. So, SmartFrog executes PrepareServer and PrepareTestsAtServer in one block, which corresponds to execution of PrepareTestsAtClient component. When PrepareTestsAtServer and PrepareTestsAtCleint are finished, SmartFrog proceeds to another step of execution. It run StartServer component at Server, and RunTestsAtClient component at Client. These components run in parallel and start at the same time. If any of the components fails, the whole system stops execution and the test is marked as failed. 28

34 6. Task formulation and test automation implementation Most of the components, except StartServer use Groovy scripts in which is written the sequence of actions. The component StartServer extends Fuse component, which can start and stop JBoss Fuse, with some settings like which version of Java to use. I created it during the work on this thesis. Groovy is an object-oriented programming language designed for the Java platform as a addition to the Java language with the possibilities of Python, Ruby and Smalltalk. Now, let s take a look at each component: PrepareServer: PrepareServer script deletes old JBoss Fuse instance from server and installs a new one. Then it adds the user which can access JBoss Fuse. PrepareTestsAtServer: PrepareTestsAtServer builds the specified tested service with maven, using mvn-bundle-plugin, as each tested service is an OSGi-bundle. OSGi-bundles are described in the Subsection of the Chapter 3. After the bundle was built, the component either deploys a feature, if it is present in the resource directory of the bundle, or deploys the jar archive with the bundle itself. The deployment happens through hot deploy mechanism, which means the artifact (a bundle or a feature) is copied into the deploy directory of JBoss Fuse. After the server will be started from the StartServer component, JBoss Fuse will try to start all bundles which are in deploy directory. PrepareTestsAtClient: PrepareTestsAtClient script deletes old PerfCake instance from server and installs a new one. Then it downloads the activemq.jar to the lib directory of PerfCake to be able to run tests for ActiveMQ client through JMS. StartServer: This is Java-based component which starts JBoss Fuse, and stops it when the component is terminated from RunTestsAtClient component. RunTestsAtClient: RunTestsAtClient script specifies which version of Java to use to run PerfCake. After that it runs PerfCake for the specified test scenario. 4) When SmartFrog finishes the execution, runs another Linux script in Jenkins, which collects and saves the results. The results are stored in comma-separated values (CSV) format. 29

35 6. Task formulation and test automation implementation 6.2 Environment characteristics The testing was performed on server and client machine with the following characteristics: CPU 4x Intel Xeon CPU (16 cores) Memory 36G OS Red Hat Enterprise Linux Server release 6.1 (Santiago) JVM Oracle JDK x86 64 Table 6.1: Client and Server hosts 30